2025-12-04T09:22:17.9298449Z Current runner version: '2.330.0' 2025-12-04T09:22:17.9303662Z Runner name: 'i-0c5ea43668efc70fe' 2025-12-04T09:22:17.9304310Z Runner group name: 'default' 2025-12-04T09:22:17.9305120Z Machine name: 'ip-10-0-39-147' 2025-12-04T09:22:17.9307418Z ##[group]GITHUB_TOKEN Permissions 2025-12-04T09:22:17.9309534Z Contents: read 2025-12-04T09:22:17.9310075Z Metadata: read 2025-12-04T09:22:17.9310477Z ##[endgroup] 2025-12-04T09:22:17.9312267Z Secret source: Actions 2025-12-04T09:22:17.9312859Z Prepare workflow directory 2025-12-04T09:22:17.9732084Z Prepare all required actions 2025-12-04T09:22:17.9766074Z Getting action download info 2025-12-04T09:22:18.3364631Z Download action repository 'pytorch/test-infra@main' (SHA:39aa74d619174326f4e2fb0e216151c2f29d9ffd) 2025-12-04T09:22:20.9387869Z Download action repository 'pytorch/pytorch@main' (SHA:7716da9fb23f27a65b41f9f016a2afadf281c18f) 2025-12-04T09:22:38.0059367Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-12-04T09:22:38.4187915Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-12-04T09:22:38.6841835Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-12-04T09:22:38.8773619Z Download action repository 'seemethere/download-artifact-s3@1da556a7aa0a088e3153970611f6c432d58e80e6' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-12-04T09:22:39.1315198Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-12-04T09:22:39.3910774Z Getting action download info 2025-12-04T09:22:39.5158003Z Download action repository 'actions/checkout@v4' (SHA:34e114876b0b11c390a56381ad16ebd13914f8d5) 2025-12-04T09:22:39.8523141Z Getting action download info 2025-12-04T09:22:39.9835739Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-12-04T09:22:40.2697232Z Getting action download info 2025-12-04T09:22:40.4123721Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-12-04T09:22:40.6020301Z Getting action download info 2025-12-04T09:22:40.7585368Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32) 2025-12-04T09:22:40.7588725Z ##[group] Inputs 2025-12-04T09:22:40.7589123Z build-environment: linux-jammy-py3.10-gcc11-build 2025-12-04T09:22:40.7595378Z test-matrix: {"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}]} 2025-12-04T09:22:40.7602456Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:40.7603164Z sync-tag: 2025-12-04T09:22:40.7604012Z timeout-minutes: 240 2025-12-04T09:22:40.7604307Z use-gha: 2025-12-04T09:22:40.7604510Z dashboard-tag: 2025-12-04T09:22:40.7604808Z s3-bucket: gha-artifacts 2025-12-04T09:22:40.7605085Z aws-role-to-assume: 2025-12-04T09:22:40.7605536Z disable-monitor: false 2025-12-04T09:22:40.7605871Z monitor-log-interval: 5 2025-12-04T09:22:40.7606162Z monitor-data-collect-interval: 1 2025-12-04T09:22:40.7606584Z ##[endgroup] 2025-12-04T09:22:40.7607054Z Complete job name: periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:22:40.8068868Z A job started hook has been configured by the self-hosted runner administrator 2025-12-04T09:22:40.8238548Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-12-04T09:22:40.8246075Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:40.8246477Z ##[endgroup] 2025-12-04T09:22:41.7942548Z Runner Type: linux.8xlarge.amx 2025-12-04T09:22:41.7943027Z Instance Type: m7i-flex.8xlarge 2025-12-04T09:22:41.7943246Z AMI Name: unknown 2025-12-04T09:22:41.7968032Z AMI ID: ami-08982f1c5bf93d976 2025-12-04T09:22:46.0417664Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-12-04T09:22:46.0417981Z with: 2025-12-04T09:22:46.0418506Z github-secret: *** 2025-12-04T09:22:46.0418958Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-12-04T09:22:46.0419418Z activate-with-label: false 2025-12-04T09:22:46.0419618Z label: with-ssh 2025-12-04T09:22:46.0419793Z remove-existing-keys: true 2025-12-04T09:22:46.0420201Z fail-silently: true 2025-12-04T09:22:46.0420374Z env: 2025-12-04T09:22:46.0420523Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:46.0420716Z ##[endgroup] 2025-12-04T09:22:46.1359601Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-12-04T09:22:46.1361170Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-12-04T09:22:46.1476178Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-12-04T09:22:46.1476477Z with: 2025-12-04T09:22:46.1476650Z no-sudo: true 2025-12-04T09:22:46.1476821Z submodules: recursive 2025-12-04T09:22:46.1477020Z fetch-depth: 0 2025-12-04T09:22:46.1477191Z env: 2025-12-04T09:22:46.1477387Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:46.1477580Z ##[endgroup] 2025-12-04T09:22:46.1537573Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:22:46.1538403Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:22:46.1549136Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:46.1549419Z env: 2025-12-04T09:22:46.1549639Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:46.1549854Z ##[endgroup] 2025-12-04T09:22:46.1632532Z ##[group]Run # Use all available CPUs for fetching 2025-12-04T09:22:46.1632846Z # Use all available CPUs for fetching 2025-12-04T09:22:46.1633094Z cd "${GITHUB_WORKSPACE}" 2025-12-04T09:22:46.1633317Z git config --global fetch.parallel 0 2025-12-04T09:22:46.1633570Z git config --global submodule.fetchJobs 0 2025-12-04T09:22:46.1633787Z  2025-12-04T09:22:46.1634107Z # Clean workspace. The default checkout action should also do this, but 2025-12-04T09:22:46.1634416Z # do it here as well just in case 2025-12-04T09:22:46.1634631Z if [[ -d .git ]]; then 2025-12-04T09:22:46.1634842Z  if [ -z "${NO_SUDO}" ]; then 2025-12-04T09:22:46.1635043Z  sudo git clean -ffdx 2025-12-04T09:22:46.1635234Z  else 2025-12-04T09:22:46.1635400Z  git clean -ffdx 2025-12-04T09:22:46.1635570Z  fi 2025-12-04T09:22:46.1635722Z fi 2025-12-04T09:22:46.1639795Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:46.1640053Z env: 2025-12-04T09:22:46.1640228Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:46.1640590Z NO_SUDO: true 2025-12-04T09:22:46.1640772Z ##[endgroup] 2025-12-04T09:22:46.1744910Z ##[group]Run actions/checkout@v4 2025-12-04T09:22:46.1745142Z with: 2025-12-04T09:22:46.1745342Z ref: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:22:46.1745572Z fetch-depth: 0 2025-12-04T09:22:46.1745750Z submodules: recursive 2025-12-04T09:22:46.1745937Z show-progress: false 2025-12-04T09:22:46.1746118Z repository: pytorch/pytorch 2025-12-04T09:22:46.1746426Z token: *** 2025-12-04T09:22:46.1746589Z ssh-strict: true 2025-12-04T09:22:46.1746753Z ssh-user: git 2025-12-04T09:22:46.1746921Z persist-credentials: true 2025-12-04T09:22:46.1747328Z clean: true 2025-12-04T09:22:46.1747511Z sparse-checkout-cone-mode: true 2025-12-04T09:22:46.1747713Z fetch-tags: false 2025-12-04T09:22:46.1747882Z lfs: false 2025-12-04T09:22:46.1748046Z set-safe-directory: true 2025-12-04T09:22:46.1748235Z env: 2025-12-04T09:22:46.1748395Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:46.1748579Z ##[endgroup] 2025-12-04T09:22:46.2695194Z Syncing repository: pytorch/pytorch 2025-12-04T09:22:46.2696318Z ##[group]Getting Git version info 2025-12-04T09:22:46.2696677Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-12-04T09:22:46.2697150Z [command]/usr/bin/git version 2025-12-04T09:22:46.2697360Z git version 2.50.1 2025-12-04T09:22:46.2698080Z ##[endgroup] 2025-12-04T09:22:46.2711015Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/fb7fb64d-6648-4799-a31a-3f0f6da6b87e/.gitconfig' 2025-12-04T09:22:46.2747243Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/fb7fb64d-6648-4799-a31a-3f0f6da6b87e' before making global git config changes 2025-12-04T09:22:46.2748031Z Adding repository directory to the temporary git global config as a safe directory 2025-12-04T09:22:46.2748921Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:22:46.2794728Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-12-04T09:22:46.2796689Z ##[group]Initializing the repository 2025-12-04T09:22:46.2798386Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:22:46.2829463Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-12-04T09:22:46.2829932Z hint: is subject to change. To configure the initial branch name to use in all 2025-12-04T09:22:46.2830312Z hint: of your new repositories, which will suppress this warning, call: 2025-12-04T09:22:46.2830571Z hint: 2025-12-04T09:22:46.2830817Z hint: git config --global init.defaultBranch 2025-12-04T09:22:46.2831050Z hint: 2025-12-04T09:22:46.2831270Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-12-04T09:22:46.2831628Z hint: 'development'. The just-created branch can be renamed via this command: 2025-12-04T09:22:46.2831902Z hint: 2025-12-04T09:22:46.2832059Z hint: git branch -m 2025-12-04T09:22:46.2832235Z hint: 2025-12-04T09:22:46.2832502Z hint: Disable this message with "git config set advice.defaultBranchName false" 2025-12-04T09:22:46.2832926Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-12-04T09:22:46.2839683Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-12-04T09:22:46.2867550Z ##[endgroup] 2025-12-04T09:22:46.2867906Z ##[group]Disabling automatic garbage collection 2025-12-04T09:22:46.2877732Z [command]/usr/bin/git config --local gc.auto 0 2025-12-04T09:22:46.2911102Z ##[endgroup] 2025-12-04T09:22:46.2911492Z ##[group]Setting up auth 2025-12-04T09:22:46.2911868Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-12-04T09:22:46.2938366Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-12-04T09:22:46.3273131Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-12-04T09:22:46.3305035Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-12-04T09:22:46.3602072Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:22:46.3634370Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url 2025-12-04T09:22:46.3920347Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-12-04T09:22:46.3985109Z ##[endgroup] 2025-12-04T09:22:46.3989184Z ##[group]Fetching the repository 2025-12-04T09:22:46.3995407Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-12-04T09:23:39.7312834Z From https://github.com/pytorch/pytorch 2025-12-04T09:23:39.7314983Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-12-04T09:23:39.7315499Z * [new branch] 2.9.1 -> origin/2.9.1 2025-12-04T09:23:39.7315930Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-12-04T09:23:39.7316506Z * [new branch] Flamefire-patch-1 -> origin/Flamefire-patch-1 2025-12-04T09:23:39.7327041Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-12-04T09:23:39.7327612Z * [new branch] HOPrintFunc -> origin/HOPrintFunc 2025-12-04T09:23:39.7328006Z * [new branch] IvanKobzarev/stack/1 -> origin/IvanKobzarev/stack/1 2025-12-04T09:23:39.7328702Z * [new branch] NicoshevSVE128 -> origin/NicoshevSVE128 2025-12-04T09:23:39.7329113Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-12-04T09:23:39.7329549Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-12-04T09:23:39.7329962Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-12-04T09:23:39.7330352Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-12-04T09:23:39.7330723Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-12-04T09:23:39.7331149Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-12-04T09:23:39.7331514Z * [new branch] VLA_exp -> origin/VLA_exp 2025-12-04T09:23:39.7331845Z * [new branch] activation_bench -> origin/activation_bench 2025-12-04T09:23:39.7332198Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-12-04T09:23:39.7332553Z * [new branch] adi/onednn_aarch64 -> origin/adi/onednn_aarch64 2025-12-04T09:23:39.7332870Z * [new branch] adi/test -> origin/adi/test 2025-12-04T09:23:39.7333192Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-12-04T09:23:39.7333508Z * [new branch] adi/test_m8g -> origin/adi/test_m8g 2025-12-04T09:23:39.7333828Z * [new branch] adi/test_onednn -> origin/adi/test_onednn 2025-12-04T09:23:39.7334159Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-12-04T09:23:39.7334513Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-12-04T09:23:39.7334852Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-12-04T09:23:39.7335210Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-12-04T09:23:39.7335578Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-12-04T09:23:39.7335998Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-12-04T09:23:39.7336490Z * [new branch] albanD-patch-1 -> origin/albanD-patch-1 2025-12-04T09:23:39.7336858Z * [new branch] also-surround-shimh -> origin/also-surround-shimh 2025-12-04T09:23:39.7337226Z * [new branch] angelayi/aot_compile -> origin/angelayi/aot_compile 2025-12-04T09:23:39.7337658Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-12-04T09:23:39.7338084Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-12-04T09:23:39.7338519Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-12-04T09:23:39.7338957Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-12-04T09:23:39.7339352Z * [new branch] angelayi/inductor_const -> origin/angelayi/inductor_const 2025-12-04T09:23:39.7339732Z * [new branch] angelayi/lstm -> origin/angelayi/lstm 2025-12-04T09:23:39.7340353Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-12-04T09:23:39.7341216Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-12-04T09:23:39.7341958Z * [new branch] angelayi/side_eff -> origin/angelayi/side_eff 2025-12-04T09:23:39.7343757Z * [new branch] angelayi/state_dict -> origin/angelayi/state_dict 2025-12-04T09:23:39.7344352Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-12-04T09:23:39.7344767Z * [new branch] angelayi/symm_mem -> origin/angelayi/symm_mem 2025-12-04T09:23:39.7345134Z * [new branch] angelayi/test_cpp -> origin/angelayi/test_cpp 2025-12-04T09:23:39.7345520Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-12-04T09:23:39.7346339Z * [new branch] annotate_assert -> origin/annotate_assert 2025-12-04T09:23:39.7347274Z * [new branch] annotate_fallback_kernel -> origin/annotate_fallback_kernel 2025-12-04T09:23:39.7348230Z * [new branch] annotation_deepcopy -> origin/annotation_deepcopy 2025-12-04T09:23:39.7348978Z * [new branch] annotation_dynamo -> origin/annotation_dynamo 2025-12-04T09:23:39.7349760Z * [new branch] aot_eager_stack_trace -> origin/aot_eager_stack_trace 2025-12-04T09:23:39.7350496Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-12-04T09:23:39.7356701Z * [new branch] aoti_const_device -> origin/aoti_const_device 2025-12-04T09:23:39.7357139Z * [new branch] aoti_fqn_name_interface -> origin/aoti_fqn_name_interface 2025-12-04T09:23:39.7357547Z * [new branch] aoti_package_weights_binary -> origin/aoti_package_weights_binary 2025-12-04T09:23:39.7357972Z * [new branch] aoti_target_windows -> origin/aoti_target_windows 2025-12-04T09:23:39.7358433Z * [new branch] arsh/feat/inductor_check_profiling -> origin/arsh/feat/inductor_check_profiling 2025-12-04T09:23:39.7358841Z * [new branch] async_tp -> origin/async_tp 2025-12-04T09:23:39.7359235Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-12-04T09:23:39.7359710Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-12-04T09:23:39.7360132Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2025-12-04T09:23:39.7360821Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-12-04T09:23:39.7361166Z * [new branch] atalman-patch-4 -> origin/atalman-patch-4 2025-12-04T09:23:39.7362029Z * [new branch] atalman-patch-5 -> origin/atalman-patch-5 2025-12-04T09:23:39.7362507Z * [new branch] atalman-patch-6 -> origin/atalman-patch-6 2025-12-04T09:23:39.7362959Z * [new branch] atalman-patch-7 -> origin/atalman-patch-7 2025-12-04T09:23:39.7363334Z * [new branch] atalman-patch-8 -> origin/atalman-patch-8 2025-12-04T09:23:39.7364620Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-12-04T09:23:39.7365180Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-12-04T09:23:39.7365716Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-12-04T09:23:39.7366275Z * [new branch] attention_benchmarking_clean -> origin/attention_benchmarking_clean 2025-12-04T09:23:39.7367723Z * [new branch] bahuang/dt_fix_scalar_add -> origin/bahuang/dt_fix_scalar_add 2025-12-04T09:23:39.7368169Z * [new branch] bahuang/fix_debug_mode -> origin/bahuang/fix_debug_mode 2025-12-04T09:23:39.7368666Z * [new branch] bahuang/fix_expand -> origin/bahuang/fix_expand 2025-12-04T09:23:39.7369409Z * [new branch] bahuang/test -> origin/bahuang/test 2025-12-04T09:23:39.7370549Z * [new branch] base/1.5 -> origin/base/1.5 2025-12-04T09:23:39.7375536Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-12-04T09:23:39.7376017Z * [new branch] bench_scaled_mm_ops -> origin/bench_scaled_mm_ops 2025-12-04T09:23:39.7376630Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-12-04T09:23:39.7377016Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-12-04T09:23:39.7377390Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-12-04T09:23:39.7377752Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-12-04T09:23:39.7378104Z * [new branch] bf/bug-static-input -> origin/bf/bug-static-input 2025-12-04T09:23:39.7378450Z * [new branch] bf/cg-backend -> origin/bf/cg-backend 2025-12-04T09:23:39.7378788Z * [new branch] bf/cg-nccl-test -> origin/bf/cg-nccl-test 2025-12-04T09:23:39.7379134Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-12-04T09:23:39.7379649Z * [new branch] bf/clean-torchbench-hf -> origin/bf/clean-torchbench-hf 2025-12-04T09:23:39.7380299Z * [new branch] bf/combo-debug-log -> origin/bf/combo-debug-log 2025-12-04T09:23:39.7381021Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-12-04T09:23:39.7382390Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-12-04T09:23:39.7383115Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-12-04T09:23:39.7387282Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-12-04T09:23:39.7388205Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-12-04T09:23:39.7388701Z * [new branch] bf/dynamo-partition -> origin/bf/dynamo-partition 2025-12-04T09:23:39.7389059Z * [new branch] bf/lite -> origin/bf/lite 2025-12-04T09:23:39.7389447Z * [new branch] bf/pa-non-divisible -> origin/bf/pa-non-divisible 2025-12-04T09:23:39.7389880Z * [new branch] bf/partition-cache-free-symbols -> origin/bf/partition-cache-free-symbols 2025-12-04T09:23:39.7390341Z * [new branch] bf/partition-memory-plan -> origin/bf/partition-memory-plan 2025-12-04T09:23:39.7392314Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-12-04T09:23:39.7392745Z * [new branch] bf/partition-view-fallback -> origin/bf/partition-view-fallback 2025-12-04T09:23:39.7395772Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-12-04T09:23:39.7396693Z * [new branch] bf/timm-nov-26-2025 -> origin/bf/timm-nov-26-2025 2025-12-04T09:23:39.7397171Z * [new branch] bf/transformer-pin-4-57-3 -> origin/bf/transformer-pin-4-57-3 2025-12-04T09:23:39.7397619Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-12-04T09:23:39.7398064Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-12-04T09:23:39.7398467Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-12-04T09:23:39.7398882Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-12-04T09:23:39.7399271Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-12-04T09:23:39.7399702Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-12-04T09:23:39.7400463Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-12-04T09:23:39.7400902Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-12-04T09:23:39.7401315Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-12-04T09:23:39.7401840Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-12-04T09:23:39.7402225Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-12-04T09:23:39.7402633Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-12-04T09:23:39.7403043Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-12-04T09:23:39.7403471Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-12-04T09:23:39.7403882Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-12-04T09:23:39.7404284Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-12-04T09:23:39.7408377Z * [new branch] brister/fx_device_type -> origin/brister/fx_device_type 2025-12-04T09:23:39.7408809Z * [new branch] brister/test_inductor_all_fx -> origin/brister/test_inductor_all_fx 2025-12-04T09:23:39.7409290Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-12-04T09:23:39.7409723Z * [new branch] bwd-backup -> origin/bwd-backup 2025-12-04T09:23:39.7410051Z * [new branch] c57382a49 -> origin/c57382a49 2025-12-04T09:23:39.7410372Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-12-04T09:23:39.7410699Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-12-04T09:23:39.7411080Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-12-04T09:23:39.7411858Z * [new branch] cccclai-patch-1 -> origin/cccclai-patch-1 2025-12-04T09:23:39.7412528Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7413085Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7414291Z * [new branch] cherry-pick-162208-by-pytorch_bot_bot_ -> origin/cherry-pick-162208-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7416557Z * [new branch] cherry-pick-163169-by-pytorch_bot_bot_ -> origin/cherry-pick-163169-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7417154Z * [new branch] cherry-pick-165086-by-pytorch_bot_bot_ -> origin/cherry-pick-165086-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7417695Z * [new branch] cherry-pick-165514-by-pytorch_bot_bot_ -> origin/cherry-pick-165514-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7418196Z * [new branch] cherry-pick-165601-by-pytorch_bot_bot_ -> origin/cherry-pick-165601-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7418774Z * [new branch] cherry-pick-165667-by-pytorch_bot_bot_ -> origin/cherry-pick-165667-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7419285Z * [new branch] cherry-pick-165815-by-pytorch_bot_bot_ -> origin/cherry-pick-165815-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7419810Z * [new branch] cherry-pick-165922-by-pytorch_bot_bot_ -> origin/cherry-pick-165922-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7420387Z * [new branch] cherry-pick-166148-by-pytorch_bot_bot_ -> origin/cherry-pick-166148-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7421158Z * [new branch] cherry-pick-166181-by-pytorch_bot_bot_ -> origin/cherry-pick-166181-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7421887Z * [new branch] cherry-pick-166404-by-pytorch_bot_bot_ -> origin/cherry-pick-166404-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7422732Z * [new branch] cherry-pick-166427-by-pytorch_bot_bot_ -> origin/cherry-pick-166427-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7423648Z * [new branch] cherry-pick-166480-by-pytorch_bot_bot_ -> origin/cherry-pick-166480-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7424552Z * [new branch] cherry-pick-166570-by-pytorch_bot_bot_ -> origin/cherry-pick-166570-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7426057Z * [new branch] cherry-pick-166993-by-pytorch_bot_bot_ -> origin/cherry-pick-166993-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7426606Z * [new branch] cherry-pick-167111-by-pytorch_bot_bot_ -> origin/cherry-pick-167111-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7427110Z * [new branch] cherry-pick-167478-by-pytorch_bot_bot_ -> origin/cherry-pick-167478-by-pytorch_bot_bot_ 2025-12-04T09:23:39.7427578Z * [new branch] cherry_pick_166036_166040 -> origin/cherry_pick_166036_166040 2025-12-04T09:23:39.7427966Z * [new branch] cherry_pick_166457 -> origin/cherry_pick_166457 2025-12-04T09:23:39.7429052Z * [new branch] cherrypick_166338 -> origin/cherrypick_166338 2025-12-04T09:23:39.7429605Z * [new branch] cherrypick_166458 -> origin/cherrypick_166458 2025-12-04T09:23:39.7430383Z * [new branch] cherrypick_166586 -> origin/cherrypick_166586 2025-12-04T09:23:39.7431138Z * [new branch] cherrypick_166956 -> origin/cherrypick_166956 2025-12-04T09:23:39.7431898Z * [new branch] ci_attn -> origin/ci_attn 2025-12-04T09:23:39.7432639Z * [new branch] codex-testing -> origin/codex-testing 2025-12-04T09:23:39.7434135Z * [new branch] codex/add-check_memory_overlap-helper-functions -> origin/codex/add-check_memory_overlap-helper-functions 2025-12-04T09:23:39.7434712Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-12-04T09:23:39.7436812Z * [new branch] codex/investigate-segfaults-in-get_tensor_storage_id -> origin/codex/investigate-segfaults-in-get_tensor_storage_id 2025-12-04T09:23:39.7437527Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-12-04T09:23:39.7438051Z * [new branch] compatiblpy39util -> origin/compatiblpy39util 2025-12-04T09:23:39.7438416Z * [new branch] cond_hop_device -> origin/cond_hop_device 2025-12-04T09:23:39.7438748Z * [new branch] context_test -> origin/context_test 2025-12-04T09:23:39.7440022Z * [new branch] copilot/code-style-cleanup-python-pip -> origin/copilot/code-style-cleanup-python-pip 2025-12-04T09:23:39.7440884Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-12-04T09:23:39.7441810Z * [new branch] cpp-docs-dependency-upgrade -> origin/cpp-docs-dependency-upgrade 2025-12-04T09:23:39.7443229Z * [new branch] crpa/typo-in-inductor_comm_lowering -> origin/crpa/typo-in-inductor_comm_lowering 2025-12-04T09:23:39.7444007Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-12-04T09:23:39.7444730Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-12-04T09:23:39.7445471Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-12-04T09:23:39.7446286Z * [new branch] csl/clean_up -> origin/csl/clean_up 2025-12-04T09:23:39.7447783Z * [new branch] csl/fix_retry_segfault_exit -> origin/csl/fix_retry_segfault_exit 2025-12-04T09:23:39.7448164Z * [new branch] csl/katex -> origin/csl/katex 2025-12-04T09:23:39.7449552Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-12-04T09:23:39.7453117Z * [new branch] csl/lint_testing -> origin/csl/lint_testing 2025-12-04T09:23:39.7453544Z * [new branch] csl/lint_thing -> origin/csl/lint_thing 2025-12-04T09:23:39.7454189Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-12-04T09:23:39.7454597Z * [new branch] csl/manually_gen_json -> origin/csl/manually_gen_json 2025-12-04T09:23:39.7454974Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-12-04T09:23:39.7457763Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-12-04T09:23:39.7458139Z * [new branch] csl/print_timing -> origin/csl/print_timing 2025-12-04T09:23:39.7458515Z * [new branch] csl/remove_experiment -> origin/csl/remove_experiment 2025-12-04T09:23:39.7458950Z * [new branch] csl/remove_maybe_unused_var -> origin/csl/remove_maybe_unused_var 2025-12-04T09:23:39.7459402Z * [new branch] csl/remove_repo_specific_autolabel -> origin/csl/remove_repo_specific_autolabel 2025-12-04T09:23:39.7459845Z * [new branch] csl/remove_run_parallel -> origin/csl/remove_run_parallel 2025-12-04T09:23:39.7460348Z * [new branch] csl/remove_unused_vars -> origin/csl/remove_unused_vars 2025-12-04T09:23:39.7462798Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-12-04T09:23:39.7463620Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-12-04T09:23:39.7464064Z * [new branch] csl/smaller_avx_amx_runenrs -> origin/csl/smaller_avx_amx_runenrs 2025-12-04T09:23:39.7464449Z * [new branch] csl/td_job_level -> origin/csl/td_job_level 2025-12-04T09:23:39.7464917Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-12-04T09:23:39.7465398Z * [new branch] csl/test_owners_autograd_dispatch_nn -> origin/csl/test_owners_autograd_dispatch_nn 2025-12-04T09:23:39.7465899Z * [new branch] csl/test_owners_higher_confidence -> origin/csl/test_owners_higher_confidence 2025-12-04T09:23:39.7466493Z * [new branch] csl/upload_json_running -> origin/csl/upload_json_running 2025-12-04T09:23:39.7466937Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-12-04T09:23:39.7467502Z * [new branch] csl/xml_stuff -> origin/csl/xml_stuff 2025-12-04T09:23:39.7467844Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-12-04T09:23:39.7468175Z * [new branch] cuda_mempool -> origin/cuda_mempool 2025-12-04T09:23:39.7468588Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-12-04T09:23:39.7473869Z * [new branch] d4l3k/debug_plane_frtrace -> origin/d4l3k/debug_plane_frtrace 2025-12-04T09:23:39.7475844Z * [new branch] daxia6/2.8o3 -> origin/daxia6/2.8o3 2025-12-04T09:23:39.7476200Z * [new branch] debug-guard -> origin/debug-guard 2025-12-04T09:23:39.7476569Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-12-04T09:23:39.7477180Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 2025-12-04T09:23:39.7477981Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 2025-12-04T09:23:39.7478718Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-12-04T09:23:39.7479359Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-12-04T09:23:39.7479807Z * [new branch] dev/dhruva/flex_attn_opt -> origin/dev/dhruva/flex_attn_opt 2025-12-04T09:23:39.7480196Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-12-04T09:23:39.7480877Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-12-04T09:23:39.7485493Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-12-04T09:23:39.7487416Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-12-04T09:23:39.7487939Z * [new branch] dev/joona/fix_sdpa_memtest -> origin/dev/joona/fix_sdpa_memtest 2025-12-04T09:23:39.7493891Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-12-04T09:23:39.7498715Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-12-04T09:23:39.7502586Z * [new branch] dev/joona/scalar_clamp -> origin/dev/joona/scalar_clamp 2025-12-04T09:23:39.7507585Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-12-04T09:23:39.7509668Z * [new branch] dev/joona/sdpa_api -> origin/dev/joona/sdpa_api 2025-12-04T09:23:39.7510040Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-12-04T09:23:39.7510946Z * [new branch] dev/joona/ulpAssertClose -> origin/dev/joona/ulpAssertClose 2025-12-04T09:23:39.7511577Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-12-04T09:23:39.7511972Z * [new branch] disp_counter -> origin/disp_counter 2025-12-04T09:23:39.7512362Z * [new branch] divyanshk-patch-1 -> origin/divyanshk-patch-1 2025-12-04T09:23:39.7512741Z * [new branch] docs -> origin/docs 2025-12-04T09:23:39.7513086Z * [new branch] documentation -> origin/documentation 2025-12-04T09:23:39.7513476Z * [new branch] eager_model_benchmarks -> origin/eager_model_benchmarks 2025-12-04T09:23:39.7513914Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-12-04T09:23:39.7514343Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-12-04T09:23:39.7514748Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-12-04T09:23:39.7515112Z * [new branch] eqy-patch-1 -> origin/eqy-patch-1 2025-12-04T09:23:39.7515441Z * [new branch] eqy-patch-2 -> origin/eqy-patch-2 2025-12-04T09:23:39.7515762Z * [new branch] eqy-patch-3 -> origin/eqy-patch-3 2025-12-04T09:23:39.7516093Z * [new branch] eqy-patch-4 -> origin/eqy-patch-4 2025-12-04T09:23:39.7516400Z * [new branch] eqy-patch-5 -> origin/eqy-patch-5 2025-12-04T09:23:39.7516724Z * [new branch] eqy-patch-6 -> origin/eqy-patch-6 2025-12-04T09:23:39.7517078Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-12-04T09:23:39.7517541Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-12-04T09:23:39.7518012Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-12-04T09:23:39.7518484Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-12-04T09:23:39.7519041Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-12-04T09:23:39.7519785Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-12-04T09:23:39.7520744Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-12-04T09:23:39.7521271Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-12-04T09:23:39.7521783Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-12-04T09:23:39.7522258Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-12-04T09:23:39.7522693Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-12-04T09:23:39.7523198Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-12-04T09:23:39.7523702Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-12-04T09:23:39.7524135Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-12-04T09:23:39.7524622Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-12-04T09:23:39.7525134Z * [new branch] exclamaforte/profile-diff-algo -> origin/exclamaforte/profile-diff-algo 2025-12-04T09:23:39.7525645Z * [new branch] exclamaforte/profiler-visualization -> origin/exclamaforte/profiler-visualization 2025-12-04T09:23:39.7526153Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-12-04T09:23:39.7526650Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-12-04T09:23:39.7527181Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-12-04T09:23:39.7527618Z * [new branch] exec -> origin/exec 2025-12-04T09:23:39.7527940Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-12-04T09:23:39.7528294Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-12-04T09:23:39.7528629Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-12-04T09:23:39.7528958Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-12-04T09:23:39.7529275Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-12-04T09:23:39.7529597Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-12-04T09:23:39.7529919Z * [new branch] export-D80823877 -> origin/export-D80823877 2025-12-04T09:23:39.7530236Z * [new branch] export-D80958642 -> origin/export-D80958642 2025-12-04T09:23:39.7530548Z * [new branch] export-D81054193 -> origin/export-D81054193 2025-12-04T09:23:39.7530877Z * [new branch] export-D81204584 -> origin/export-D81204584 2025-12-04T09:23:39.7531201Z * [new branch] export-D81429090 -> origin/export-D81429090 2025-12-04T09:23:39.7531522Z * [new branch] export-D82250826 -> origin/export-D82250826 2025-12-04T09:23:39.7531840Z * [new branch] export-D82253817 -> origin/export-D82253817 2025-12-04T09:23:39.7532160Z * [new branch] export-D83541846 -> origin/export-D83541846 2025-12-04T09:23:39.7532481Z * [new branch] export-D83627170 -> origin/export-D83627170 2025-12-04T09:23:39.7532797Z * [new branch] export-D83766701 -> origin/export-D83766701 2025-12-04T09:23:39.7533118Z * [new branch] export-D83768878 -> origin/export-D83768878 2025-12-04T09:23:39.7533444Z * [new branch] export-D83769447 -> origin/export-D83769447 2025-12-04T09:23:39.7533813Z * [new branch] export-D84089824 -> origin/export-D84089824 2025-12-04T09:23:39.7534128Z * [new branch] export-D84213020 -> origin/export-D84213020 2025-12-04T09:23:39.7534519Z * [new branch] export-D84373821 -> origin/export-D84373821 2025-12-04T09:23:39.7535347Z * [new branch] export-D84612194 -> origin/export-D84612194 2025-12-04T09:23:39.7536098Z * [new branch] export-D84890985 -> origin/export-D84890985 2025-12-04T09:23:39.7536802Z * [new branch] export-D85122326 -> origin/export-D85122326 2025-12-04T09:23:39.7537771Z * [new branch] export-D86256198 -> origin/export-D86256198 2025-12-04T09:23:39.7538417Z * [new branch] export-D86460608 -> origin/export-D86460608 2025-12-04T09:23:39.7539435Z * [new branch] export-D86474796 -> origin/export-D86474796 2025-12-04T09:23:39.7540221Z * [new branch] export-D86712396 -> origin/export-D86712396 2025-12-04T09:23:39.7540833Z * [new branch] export-D87022129 -> origin/export-D87022129 2025-12-04T09:23:39.7541653Z * [new branch] export-D87838959 -> origin/export-D87838959 2025-12-04T09:23:39.7543422Z * [new branch] export-D88319437 -> origin/export-D88319437 2025-12-04T09:23:39.7543943Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-12-04T09:23:39.7544368Z * [new branch] ezyang-titan-october -> origin/ezyang-titan-october 2025-12-04T09:23:39.7544798Z * [new branch] ezyang-titan-october2 -> origin/ezyang-titan-october2 2025-12-04T09:23:39.7545332Z * [new branch] ezyang-war -> origin/ezyang-war 2025-12-04T09:23:39.7547615Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-12-04T09:23:39.7548092Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-12-04T09:23:39.7548703Z * [new branch] fadeputr/sequence_fbgemm -> origin/fadeputr/sequence_fbgemm 2025-12-04T09:23:39.7549326Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-12-04T09:23:39.7553228Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-12-04T09:23:39.7553591Z * [new branch] fca -> origin/fca 2025-12-04T09:23:39.7553907Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-12-04T09:23:39.7554214Z * [new branch] fca5 -> origin/fca5 2025-12-04T09:23:39.7554571Z * [new branch] feature/justknobs-cpp -> origin/feature/justknobs-cpp 2025-12-04T09:23:39.7554976Z * [new branch] feature/numa-forkserver -> origin/feature/numa-forkserver 2025-12-04T09:23:39.7555670Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-12-04T09:23:39.7556359Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-12-04T09:23:39.7557942Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-12-04T09:23:39.7559232Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-12-04T09:23:39.7559616Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-12-04T09:23:39.7559979Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-12-04T09:23:39.7560809Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-12-04T09:23:39.7564515Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-12-04T09:23:39.7565085Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-12-04T09:23:39.7565465Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-12-04T09:23:39.7565865Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-12-04T09:23:39.7566323Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-12-04T09:23:39.7566673Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-12-04T09:23:39.7567004Z * [new branch] fix_addmm_issue -> origin/fix_addmm_issue 2025-12-04T09:23:39.7568259Z * [new branch] fix_amd_missing_cluster_dims -> origin/fix_amd_missing_cluster_dims 2025-12-04T09:23:39.7568745Z * [new branch] fix_bench_bwd_pass -> origin/fix_bench_bwd_pass 2025-12-04T09:23:39.7574088Z * [new branch] fix_mem_profiler_config -> origin/fix_mem_profiler_config 2025-12-04T09:23:39.7579059Z * [new branch] fix_nvrtc_discovery -> origin/fix_nvrtc_discovery 2025-12-04T09:23:39.7583318Z * [new branch] fix_op_runner -> origin/fix_op_runner 2025-12-04T09:23:39.7588346Z * [new branch] fix_ubn_159469 -> origin/fix_ubn_159469 2025-12-04T09:23:39.7590347Z * [new branch] fixes-triage -> origin/fixes-triage 2025-12-04T09:23:39.7590718Z * [new branch] fixflashinfer -> origin/fixflashinfer 2025-12-04T09:23:39.7591064Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-12-04T09:23:39.7591402Z * [new branch] flex-flash -> origin/flex-flash 2025-12-04T09:23:39.7591807Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-12-04T09:23:39.7592164Z * [new branch] flex_flash -> origin/flex_flash 2025-12-04T09:23:39.7592565Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-12-04T09:23:39.7593014Z * [new branch] fmassa/tests_comm_compute_scheduler -> origin/fmassa/tests_comm_compute_scheduler 2025-12-04T09:23:39.7593413Z * [new branch] forkserver_fix -> origin/forkserver_fix 2025-12-04T09:23:39.7593729Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-12-04T09:23:39.7594048Z * [new branch] fx_cpp -> origin/fx_cpp 2025-12-04T09:23:39.7594353Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-12-04T09:23:39.7594674Z * [new branch] galv-patch-1 -> origin/galv-patch-1 2025-12-04T09:23:39.7595090Z * [new branch] galv/cudagraphs-conditional-nodes-4 -> origin/galv/cudagraphs-conditional-nodes-4 2025-12-04T09:23:39.7595556Z * [new branch] georgehong/cmakelists-patch -> origin/georgehong/cmakelists-patch 2025-12-04T09:23:39.7595972Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-12-04T09:23:39.7596311Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-12-04T09:23:39.7596662Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-12-04T09:23:39.7597025Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-12-04T09:23:39.7597387Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-12-04T09:23:39.7597729Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-12-04T09:23:39.7598073Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-12-04T09:23:39.7598421Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-12-04T09:23:39.7598750Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-12-04T09:23:39.7599225Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-12-04T09:23:39.7599558Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-12-04T09:23:39.7599889Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-12-04T09:23:39.7600275Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-12-04T09:23:39.7600762Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-12-04T09:23:39.7601100Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-12-04T09:23:39.7601429Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-12-04T09:23:39.7601759Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-12-04T09:23:39.7602095Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-12-04T09:23:39.7602407Z * [new branch] gh/H-Huang/226/base -> origin/gh/H-Huang/226/base 2025-12-04T09:23:39.7603967Z * [new branch] gh/H-Huang/226/head -> origin/gh/H-Huang/226/head 2025-12-04T09:23:39.7604401Z * [new branch] gh/H-Huang/226/orig -> origin/gh/H-Huang/226/orig 2025-12-04T09:23:39.7604830Z * [new branch] gh/H-Huang/228/base -> origin/gh/H-Huang/228/base 2025-12-04T09:23:39.7605183Z * [new branch] gh/H-Huang/228/head -> origin/gh/H-Huang/228/head 2025-12-04T09:23:39.7605596Z * [new branch] gh/H-Huang/228/orig -> origin/gh/H-Huang/228/orig 2025-12-04T09:23:39.7609737Z * [new branch] gh/IvanKobzarev/150/base -> origin/gh/IvanKobzarev/150/base 2025-12-04T09:23:39.7610131Z * [new branch] gh/IvanKobzarev/150/head -> origin/gh/IvanKobzarev/150/head 2025-12-04T09:23:39.7610518Z * [new branch] gh/IvanKobzarev/150/orig -> origin/gh/IvanKobzarev/150/orig 2025-12-04T09:23:39.7610898Z * [new branch] gh/IvanKobzarev/157/base -> origin/gh/IvanKobzarev/157/base 2025-12-04T09:23:39.7611274Z * [new branch] gh/IvanKobzarev/157/head -> origin/gh/IvanKobzarev/157/head 2025-12-04T09:23:39.7611645Z * [new branch] gh/IvanKobzarev/157/orig -> origin/gh/IvanKobzarev/157/orig 2025-12-04T09:23:39.7614640Z * [new branch] gh/IvanKobzarev/159/base -> origin/gh/IvanKobzarev/159/base 2025-12-04T09:23:39.7615026Z * [new branch] gh/IvanKobzarev/159/head -> origin/gh/IvanKobzarev/159/head 2025-12-04T09:23:39.7615399Z * [new branch] gh/IvanKobzarev/159/orig -> origin/gh/IvanKobzarev/159/orig 2025-12-04T09:23:39.7615765Z * [new branch] gh/IvanKobzarev/162/base -> origin/gh/IvanKobzarev/162/base 2025-12-04T09:23:39.7616137Z * [new branch] gh/IvanKobzarev/162/head -> origin/gh/IvanKobzarev/162/head 2025-12-04T09:23:39.7616514Z * [new branch] gh/IvanKobzarev/162/orig -> origin/gh/IvanKobzarev/162/orig 2025-12-04T09:23:39.7616888Z * [new branch] gh/IvanKobzarev/163/base -> origin/gh/IvanKobzarev/163/base 2025-12-04T09:23:39.7618306Z * [new branch] gh/IvanKobzarev/163/head -> origin/gh/IvanKobzarev/163/head 2025-12-04T09:23:39.7618699Z * [new branch] gh/IvanKobzarev/163/orig -> origin/gh/IvanKobzarev/163/orig 2025-12-04T09:23:39.7619076Z * [new branch] gh/IvanKobzarev/166/base -> origin/gh/IvanKobzarev/166/base 2025-12-04T09:23:39.7619451Z * [new branch] gh/IvanKobzarev/166/head -> origin/gh/IvanKobzarev/166/head 2025-12-04T09:23:39.7619819Z * [new branch] gh/IvanKobzarev/166/orig -> origin/gh/IvanKobzarev/166/orig 2025-12-04T09:23:39.7622594Z * [new branch] gh/IvanKobzarev/167/base -> origin/gh/IvanKobzarev/167/base 2025-12-04T09:23:39.7623074Z * [new branch] gh/IvanKobzarev/167/head -> origin/gh/IvanKobzarev/167/head 2025-12-04T09:23:39.7623485Z * [new branch] gh/IvanKobzarev/167/orig -> origin/gh/IvanKobzarev/167/orig 2025-12-04T09:23:39.7623850Z * [new branch] gh/IvanKobzarev/168/base -> origin/gh/IvanKobzarev/168/base 2025-12-04T09:23:39.7624262Z * [new branch] gh/IvanKobzarev/168/head -> origin/gh/IvanKobzarev/168/head 2025-12-04T09:23:39.7625902Z * [new branch] gh/IvanKobzarev/168/orig -> origin/gh/IvanKobzarev/168/orig 2025-12-04T09:23:39.7626284Z * [new branch] gh/IvanKobzarev/169/base -> origin/gh/IvanKobzarev/169/base 2025-12-04T09:23:39.7626645Z * [new branch] gh/IvanKobzarev/169/head -> origin/gh/IvanKobzarev/169/head 2025-12-04T09:23:39.7627017Z * [new branch] gh/IvanKobzarev/169/orig -> origin/gh/IvanKobzarev/169/orig 2025-12-04T09:23:39.7629579Z * [new branch] gh/IvanKobzarev/170/base -> origin/gh/IvanKobzarev/170/base 2025-12-04T09:23:39.7630061Z * [new branch] gh/IvanKobzarev/170/head -> origin/gh/IvanKobzarev/170/head 2025-12-04T09:23:39.7630460Z * [new branch] gh/IvanKobzarev/170/orig -> origin/gh/IvanKobzarev/170/orig 2025-12-04T09:23:39.7630856Z * [new branch] gh/IvanKobzarev/171/base -> origin/gh/IvanKobzarev/171/base 2025-12-04T09:23:39.7631272Z * [new branch] gh/IvanKobzarev/171/head -> origin/gh/IvanKobzarev/171/head 2025-12-04T09:23:39.7636771Z * [new branch] gh/IvanKobzarev/171/orig -> origin/gh/IvanKobzarev/171/orig 2025-12-04T09:23:39.7638354Z * [new branch] gh/IvanKobzarev/172/base -> origin/gh/IvanKobzarev/172/base 2025-12-04T09:23:39.7638779Z * [new branch] gh/IvanKobzarev/172/head -> origin/gh/IvanKobzarev/172/head 2025-12-04T09:23:39.7639179Z * [new branch] gh/IvanKobzarev/172/orig -> origin/gh/IvanKobzarev/172/orig 2025-12-04T09:23:39.7639597Z * [new branch] gh/IvanKobzarev/173/base -> origin/gh/IvanKobzarev/173/base 2025-12-04T09:23:39.7639981Z * [new branch] gh/IvanKobzarev/173/head -> origin/gh/IvanKobzarev/173/head 2025-12-04T09:23:39.7640580Z * [new branch] gh/IvanKobzarev/173/orig -> origin/gh/IvanKobzarev/173/orig 2025-12-04T09:23:39.7641027Z * [new branch] gh/IvanKobzarev/174/base -> origin/gh/IvanKobzarev/174/base 2025-12-04T09:23:39.7641414Z * [new branch] gh/IvanKobzarev/174/head -> origin/gh/IvanKobzarev/174/head 2025-12-04T09:23:39.7641806Z * [new branch] gh/IvanKobzarev/174/orig -> origin/gh/IvanKobzarev/174/orig 2025-12-04T09:23:39.7642191Z * [new branch] gh/IvanKobzarev/175/base -> origin/gh/IvanKobzarev/175/base 2025-12-04T09:23:39.7642607Z * [new branch] gh/IvanKobzarev/175/head -> origin/gh/IvanKobzarev/175/head 2025-12-04T09:23:39.7642991Z * [new branch] gh/IvanKobzarev/175/orig -> origin/gh/IvanKobzarev/175/orig 2025-12-04T09:23:39.7643433Z * [new branch] gh/IvanKobzarev/176/base -> origin/gh/IvanKobzarev/176/base 2025-12-04T09:23:39.7643828Z * [new branch] gh/IvanKobzarev/176/head -> origin/gh/IvanKobzarev/176/head 2025-12-04T09:23:39.7644564Z * [new branch] gh/IvanKobzarev/176/orig -> origin/gh/IvanKobzarev/176/orig 2025-12-04T09:23:39.7645027Z * [new branch] gh/IvanKobzarev/177/base -> origin/gh/IvanKobzarev/177/base 2025-12-04T09:23:39.7649063Z * [new branch] gh/IvanKobzarev/177/head -> origin/gh/IvanKobzarev/177/head 2025-12-04T09:23:39.7649499Z * [new branch] gh/IvanKobzarev/177/orig -> origin/gh/IvanKobzarev/177/orig 2025-12-04T09:23:39.7649912Z * [new branch] gh/IvanKobzarev/178/base -> origin/gh/IvanKobzarev/178/base 2025-12-04T09:23:39.7650343Z * [new branch] gh/IvanKobzarev/178/head -> origin/gh/IvanKobzarev/178/head 2025-12-04T09:23:39.7650991Z * [new branch] gh/IvanKobzarev/178/orig -> origin/gh/IvanKobzarev/178/orig 2025-12-04T09:23:39.7651388Z * [new branch] gh/IvanKobzarev/179/base -> origin/gh/IvanKobzarev/179/base 2025-12-04T09:23:39.7651780Z * [new branch] gh/IvanKobzarev/179/head -> origin/gh/IvanKobzarev/179/head 2025-12-04T09:23:39.7657671Z * [new branch] gh/IvanKobzarev/179/orig -> origin/gh/IvanKobzarev/179/orig 2025-12-04T09:23:39.7658174Z * [new branch] gh/IvanKobzarev/180/base -> origin/gh/IvanKobzarev/180/base 2025-12-04T09:23:39.7658576Z * [new branch] gh/IvanKobzarev/180/head -> origin/gh/IvanKobzarev/180/head 2025-12-04T09:23:39.7658963Z * [new branch] gh/IvanKobzarev/180/orig -> origin/gh/IvanKobzarev/180/orig 2025-12-04T09:23:39.7659357Z * [new branch] gh/IvanKobzarev/181/base -> origin/gh/IvanKobzarev/181/base 2025-12-04T09:23:39.7659753Z * [new branch] gh/IvanKobzarev/181/head -> origin/gh/IvanKobzarev/181/head 2025-12-04T09:23:39.7660168Z * [new branch] gh/IvanKobzarev/181/orig -> origin/gh/IvanKobzarev/181/orig 2025-12-04T09:23:39.7660552Z * [new branch] gh/IvanKobzarev/182/base -> origin/gh/IvanKobzarev/182/base 2025-12-04T09:23:39.7660923Z * [new branch] gh/IvanKobzarev/182/head -> origin/gh/IvanKobzarev/182/head 2025-12-04T09:23:39.7661299Z * [new branch] gh/IvanKobzarev/182/orig -> origin/gh/IvanKobzarev/182/orig 2025-12-04T09:23:39.7661690Z * [new branch] gh/IvanKobzarev/183/base -> origin/gh/IvanKobzarev/183/base 2025-12-04T09:23:39.7662064Z * [new branch] gh/IvanKobzarev/183/head -> origin/gh/IvanKobzarev/183/head 2025-12-04T09:23:39.7662446Z * [new branch] gh/IvanKobzarev/183/orig -> origin/gh/IvanKobzarev/183/orig 2025-12-04T09:23:39.7667077Z * [new branch] gh/IvanKobzarev/184/base -> origin/gh/IvanKobzarev/184/base 2025-12-04T09:23:39.7667743Z * [new branch] gh/IvanKobzarev/184/head -> origin/gh/IvanKobzarev/184/head 2025-12-04T09:23:39.7668588Z * [new branch] gh/IvanKobzarev/184/orig -> origin/gh/IvanKobzarev/184/orig 2025-12-04T09:23:39.7669061Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-12-04T09:23:39.7669491Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-12-04T09:23:39.7669892Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-12-04T09:23:39.7670336Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-12-04T09:23:39.7670750Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-12-04T09:23:39.7671128Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-12-04T09:23:39.7671500Z * [new branch] gh/NikhilAPatel/5/base -> origin/gh/NikhilAPatel/5/base 2025-12-04T09:23:39.7671871Z * [new branch] gh/NikhilAPatel/5/head -> origin/gh/NikhilAPatel/5/head 2025-12-04T09:23:39.7672233Z * [new branch] gh/NikhilAPatel/5/orig -> origin/gh/NikhilAPatel/5/orig 2025-12-04T09:23:39.7672606Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-12-04T09:23:39.7678848Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-12-04T09:23:39.7679313Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-12-04T09:23:39.7679674Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-12-04T09:23:39.7680036Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-12-04T09:23:39.7680469Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-12-04T09:23:39.7681023Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-12-04T09:23:39.7681369Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-12-04T09:23:39.7681765Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-12-04T09:23:39.7682246Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-12-04T09:23:39.7682651Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-12-04T09:23:39.7682992Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-12-04T09:23:39.7683340Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-12-04T09:23:39.7683670Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-12-04T09:23:39.7684128Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-12-04T09:23:39.7688820Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-12-04T09:23:39.7694116Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-12-04T09:23:39.7695007Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-12-04T09:23:39.7695385Z * [new branch] gh/PaliC/25/head -> origin/gh/PaliC/25/head 2025-12-04T09:23:39.7695753Z * [new branch] gh/PaliC/25/next -> origin/gh/PaliC/25/next 2025-12-04T09:23:39.7696094Z * [new branch] gh/PaliC/25/orig -> origin/gh/PaliC/25/orig 2025-12-04T09:23:39.7696458Z * [new branch] gh/PaliC/26/head -> origin/gh/PaliC/26/head 2025-12-04T09:23:39.7696801Z * [new branch] gh/PaliC/26/next -> origin/gh/PaliC/26/next 2025-12-04T09:23:39.7697134Z * [new branch] gh/PaliC/26/orig -> origin/gh/PaliC/26/orig 2025-12-04T09:23:39.7697476Z * [new branch] gh/PaliC/27/next -> origin/gh/PaliC/27/next 2025-12-04T09:23:39.7697825Z * [new branch] gh/PaliC/28/head -> origin/gh/PaliC/28/head 2025-12-04T09:23:39.7698166Z * [new branch] gh/PaliC/28/next -> origin/gh/PaliC/28/next 2025-12-04T09:23:39.7698536Z * [new branch] gh/PaliC/28/orig -> origin/gh/PaliC/28/orig 2025-12-04T09:23:39.7698883Z * [new branch] gh/PaliC/29/head -> origin/gh/PaliC/29/head 2025-12-04T09:23:39.7699226Z * [new branch] gh/PaliC/29/next -> origin/gh/PaliC/29/next 2025-12-04T09:23:39.7699566Z * [new branch] gh/PaliC/29/orig -> origin/gh/PaliC/29/orig 2025-12-04T09:23:39.7699930Z * [new branch] gh/PaliC/30/head -> origin/gh/PaliC/30/head 2025-12-04T09:23:39.7700333Z * [new branch] gh/PaliC/30/next -> origin/gh/PaliC/30/next 2025-12-04T09:23:39.7700691Z * [new branch] gh/PaliC/30/orig -> origin/gh/PaliC/30/orig 2025-12-04T09:23:39.7701025Z * [new branch] gh/PaliC/31/head -> origin/gh/PaliC/31/head 2025-12-04T09:23:39.7701369Z * [new branch] gh/PaliC/31/next -> origin/gh/PaliC/31/next 2025-12-04T09:23:39.7701714Z * [new branch] gh/PaliC/31/orig -> origin/gh/PaliC/31/orig 2025-12-04T09:23:39.7702113Z * [new branch] gh/PaulZhang12/25/base -> origin/gh/PaulZhang12/25/base 2025-12-04T09:23:39.7702522Z * [new branch] gh/PaulZhang12/25/head -> origin/gh/PaulZhang12/25/head 2025-12-04T09:23:39.7702906Z * [new branch] gh/PaulZhang12/25/orig -> origin/gh/PaulZhang12/25/orig 2025-12-04T09:23:39.7703283Z * [new branch] gh/PaulZhang12/28/base -> origin/gh/PaulZhang12/28/base 2025-12-04T09:23:39.7703666Z * [new branch] gh/PaulZhang12/28/head -> origin/gh/PaulZhang12/28/head 2025-12-04T09:23:39.7704192Z * [new branch] gh/PaulZhang12/28/orig -> origin/gh/PaulZhang12/28/orig 2025-12-04T09:23:39.7704575Z * [new branch] gh/PaulZhang12/31/base -> origin/gh/PaulZhang12/31/base 2025-12-04T09:23:39.7704961Z * [new branch] gh/PaulZhang12/31/head -> origin/gh/PaulZhang12/31/head 2025-12-04T09:23:39.7705405Z * [new branch] gh/PaulZhang12/31/orig -> origin/gh/PaulZhang12/31/orig 2025-12-04T09:23:39.7706065Z * [new branch] gh/PaulZhang12/37/base -> origin/gh/PaulZhang12/37/base 2025-12-04T09:23:39.7706730Z * [new branch] gh/PaulZhang12/37/head -> origin/gh/PaulZhang12/37/head 2025-12-04T09:23:39.7707124Z * [new branch] gh/PaulZhang12/37/orig -> origin/gh/PaulZhang12/37/orig 2025-12-04T09:23:39.7708466Z * [new branch] gh/PaulZhang12/40/base -> origin/gh/PaulZhang12/40/base 2025-12-04T09:23:39.7709143Z * [new branch] gh/PaulZhang12/40/head -> origin/gh/PaulZhang12/40/head 2025-12-04T09:23:39.7709626Z * [new branch] gh/PaulZhang12/40/orig -> origin/gh/PaulZhang12/40/orig 2025-12-04T09:23:39.7711249Z * [new branch] gh/PaulZhang12/42/base -> origin/gh/PaulZhang12/42/base 2025-12-04T09:23:39.7711657Z * [new branch] gh/PaulZhang12/42/head -> origin/gh/PaulZhang12/42/head 2025-12-04T09:23:39.7712411Z * [new branch] gh/PaulZhang12/43/base -> origin/gh/PaulZhang12/43/base 2025-12-04T09:23:39.7713176Z * [new branch] gh/PaulZhang12/43/head -> origin/gh/PaulZhang12/43/head 2025-12-04T09:23:39.7713867Z * [new branch] gh/PaulZhang12/43/orig -> origin/gh/PaulZhang12/43/orig 2025-12-04T09:23:39.7715214Z * [new branch] gh/PaulZhang12/44/base -> origin/gh/PaulZhang12/44/base 2025-12-04T09:23:39.7715602Z * [new branch] gh/PaulZhang12/44/head -> origin/gh/PaulZhang12/44/head 2025-12-04T09:23:39.7718269Z * [new branch] gh/PaulZhang12/45/base -> origin/gh/PaulZhang12/45/base 2025-12-04T09:23:39.7718756Z * [new branch] gh/PaulZhang12/45/head -> origin/gh/PaulZhang12/45/head 2025-12-04T09:23:39.7719155Z * [new branch] gh/PaulZhang12/45/orig -> origin/gh/PaulZhang12/45/orig 2025-12-04T09:23:39.7719549Z * [new branch] gh/PaulZhang12/46/base -> origin/gh/PaulZhang12/46/base 2025-12-04T09:23:39.7719952Z * [new branch] gh/PaulZhang12/46/head -> origin/gh/PaulZhang12/46/head 2025-12-04T09:23:39.7720710Z * [new branch] gh/PaulZhang12/46/orig -> origin/gh/PaulZhang12/46/orig 2025-12-04T09:23:39.7721667Z * [new branch] gh/PaulZhang12/47/base -> origin/gh/PaulZhang12/47/base 2025-12-04T09:23:39.7722445Z * [new branch] gh/PaulZhang12/47/head -> origin/gh/PaulZhang12/47/head 2025-12-04T09:23:39.7723171Z * [new branch] gh/PaulZhang12/47/orig -> origin/gh/PaulZhang12/47/orig 2025-12-04T09:23:39.7724409Z * [new branch] gh/PaulZhang12/48/base -> origin/gh/PaulZhang12/48/base 2025-12-04T09:23:39.7725223Z * [new branch] gh/PaulZhang12/48/head -> origin/gh/PaulZhang12/48/head 2025-12-04T09:23:39.7725851Z * [new branch] gh/PaulZhang12/48/orig -> origin/gh/PaulZhang12/48/orig 2025-12-04T09:23:39.7726604Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-12-04T09:23:39.7727320Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-12-04T09:23:39.7729394Z * [new branch] gh/SherlockNoMad/1/base -> origin/gh/SherlockNoMad/1/base 2025-12-04T09:23:39.7729863Z * [new branch] gh/SherlockNoMad/1/head -> origin/gh/SherlockNoMad/1/head 2025-12-04T09:23:39.7730720Z * [new branch] gh/SherlockNoMad/10/base -> origin/gh/SherlockNoMad/10/base 2025-12-04T09:23:39.7731137Z * [new branch] gh/SherlockNoMad/10/head -> origin/gh/SherlockNoMad/10/head 2025-12-04T09:23:39.7735477Z * [new branch] gh/SherlockNoMad/10/orig -> origin/gh/SherlockNoMad/10/orig 2025-12-04T09:23:39.7736354Z * [new branch] gh/SherlockNoMad/11/base -> origin/gh/SherlockNoMad/11/base 2025-12-04T09:23:39.7737036Z * [new branch] gh/SherlockNoMad/11/head -> origin/gh/SherlockNoMad/11/head 2025-12-04T09:23:39.7737529Z * [new branch] gh/SherlockNoMad/11/orig -> origin/gh/SherlockNoMad/11/orig 2025-12-04T09:23:39.7737987Z * [new branch] gh/SherlockNoMad/12/base -> origin/gh/SherlockNoMad/12/base 2025-12-04T09:23:39.7738411Z * [new branch] gh/SherlockNoMad/12/head -> origin/gh/SherlockNoMad/12/head 2025-12-04T09:23:39.7738805Z * [new branch] gh/SherlockNoMad/12/orig -> origin/gh/SherlockNoMad/12/orig 2025-12-04T09:23:39.7739196Z * [new branch] gh/SherlockNoMad/15/base -> origin/gh/SherlockNoMad/15/base 2025-12-04T09:23:39.7739599Z * [new branch] gh/SherlockNoMad/15/head -> origin/gh/SherlockNoMad/15/head 2025-12-04T09:23:39.7739984Z * [new branch] gh/SherlockNoMad/15/orig -> origin/gh/SherlockNoMad/15/orig 2025-12-04T09:23:39.7740376Z * [new branch] gh/SherlockNoMad/17/base -> origin/gh/SherlockNoMad/17/base 2025-12-04T09:23:39.7740772Z * [new branch] gh/SherlockNoMad/17/head -> origin/gh/SherlockNoMad/17/head 2025-12-04T09:23:39.7741486Z * [new branch] gh/SherlockNoMad/17/orig -> origin/gh/SherlockNoMad/17/orig 2025-12-04T09:23:39.7743505Z * [new branch] gh/SherlockNoMad/18/base -> origin/gh/SherlockNoMad/18/base 2025-12-04T09:23:39.7743897Z * [new branch] gh/SherlockNoMad/18/head -> origin/gh/SherlockNoMad/18/head 2025-12-04T09:23:39.7744274Z * [new branch] gh/SherlockNoMad/18/orig -> origin/gh/SherlockNoMad/18/orig 2025-12-04T09:23:39.7745610Z * [new branch] gh/SherlockNoMad/19/base -> origin/gh/SherlockNoMad/19/base 2025-12-04T09:23:39.7746320Z * [new branch] gh/SherlockNoMad/19/head -> origin/gh/SherlockNoMad/19/head 2025-12-04T09:23:39.7746966Z * [new branch] gh/SherlockNoMad/19/orig -> origin/gh/SherlockNoMad/19/orig 2025-12-04T09:23:39.7747504Z * [new branch] gh/SherlockNoMad/2/base -> origin/gh/SherlockNoMad/2/base 2025-12-04T09:23:39.7748116Z * [new branch] gh/SherlockNoMad/2/head -> origin/gh/SherlockNoMad/2/head 2025-12-04T09:23:39.7748715Z * [new branch] gh/SherlockNoMad/20/base -> origin/gh/SherlockNoMad/20/base 2025-12-04T09:23:39.7750281Z * [new branch] gh/SherlockNoMad/20/head -> origin/gh/SherlockNoMad/20/head 2025-12-04T09:23:39.7750673Z * [new branch] gh/SherlockNoMad/20/orig -> origin/gh/SherlockNoMad/20/orig 2025-12-04T09:23:39.7755638Z * [new branch] gh/SherlockNoMad/21/base -> origin/gh/SherlockNoMad/21/base 2025-12-04T09:23:39.7756111Z * [new branch] gh/SherlockNoMad/21/head -> origin/gh/SherlockNoMad/21/head 2025-12-04T09:23:39.7756510Z * [new branch] gh/SherlockNoMad/21/orig -> origin/gh/SherlockNoMad/21/orig 2025-12-04T09:23:39.7756919Z * [new branch] gh/SherlockNoMad/3/base -> origin/gh/SherlockNoMad/3/base 2025-12-04T09:23:39.7757313Z * [new branch] gh/SherlockNoMad/3/head -> origin/gh/SherlockNoMad/3/head 2025-12-04T09:23:39.7757701Z * [new branch] gh/SherlockNoMad/4/base -> origin/gh/SherlockNoMad/4/base 2025-12-04T09:23:39.7758089Z * [new branch] gh/SherlockNoMad/4/head -> origin/gh/SherlockNoMad/4/head 2025-12-04T09:23:39.7758465Z * [new branch] gh/SherlockNoMad/5/base -> origin/gh/SherlockNoMad/5/base 2025-12-04T09:23:39.7758846Z * [new branch] gh/SherlockNoMad/5/head -> origin/gh/SherlockNoMad/5/head 2025-12-04T09:23:39.7759299Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-12-04T09:23:39.7760092Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-12-04T09:23:39.7764665Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-12-04T09:23:39.7765290Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-12-04T09:23:39.7766047Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-12-04T09:23:39.7766460Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-12-04T09:23:39.7766850Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-12-04T09:23:39.7767224Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-12-04T09:23:39.7767599Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-12-04T09:23:39.7767985Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-12-04T09:23:39.7768358Z * [new branch] gh/StrongerXi/73/base -> origin/gh/StrongerXi/73/base 2025-12-04T09:23:39.7768719Z * [new branch] gh/StrongerXi/73/head -> origin/gh/StrongerXi/73/head 2025-12-04T09:23:39.7769099Z * [new branch] gh/StrongerXi/73/orig -> origin/gh/StrongerXi/73/orig 2025-12-04T09:23:39.7769859Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-12-04T09:23:39.7770388Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-12-04T09:23:39.7771057Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-12-04T09:23:39.7776895Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-12-04T09:23:39.7781889Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-12-04T09:23:39.7785740Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-12-04T09:23:39.7790752Z * [new branch] gh/XilunWu/168/base -> origin/gh/XilunWu/168/base 2025-12-04T09:23:39.7791175Z * [new branch] gh/XilunWu/168/head -> origin/gh/XilunWu/168/head 2025-12-04T09:23:39.7791534Z * [new branch] gh/XilunWu/168/orig -> origin/gh/XilunWu/168/orig 2025-12-04T09:23:39.7791887Z * [new branch] gh/XilunWu/169/base -> origin/gh/XilunWu/169/base 2025-12-04T09:23:39.7792235Z * [new branch] gh/XilunWu/169/head -> origin/gh/XilunWu/169/head 2025-12-04T09:23:39.7792571Z * [new branch] gh/XilunWu/169/orig -> origin/gh/XilunWu/169/orig 2025-12-04T09:23:39.7792912Z * [new branch] gh/XilunWu/170/base -> origin/gh/XilunWu/170/base 2025-12-04T09:23:39.7793255Z * [new branch] gh/XilunWu/170/head -> origin/gh/XilunWu/170/head 2025-12-04T09:23:39.7793601Z * [new branch] gh/XilunWu/170/orig -> origin/gh/XilunWu/170/orig 2025-12-04T09:23:39.7793935Z * [new branch] gh/XilunWu/171/base -> origin/gh/XilunWu/171/base 2025-12-04T09:23:39.7794259Z * [new branch] gh/XilunWu/171/head -> origin/gh/XilunWu/171/head 2025-12-04T09:23:39.7794592Z * [new branch] gh/XilunWu/171/orig -> origin/gh/XilunWu/171/orig 2025-12-04T09:23:39.7794923Z * [new branch] gh/XilunWu/173/base -> origin/gh/XilunWu/173/base 2025-12-04T09:23:39.7795256Z * [new branch] gh/XilunWu/173/head -> origin/gh/XilunWu/173/head 2025-12-04T09:23:39.7795598Z * [new branch] gh/XilunWu/173/orig -> origin/gh/XilunWu/173/orig 2025-12-04T09:23:39.7795932Z * [new branch] gh/XilunWu/175/base -> origin/gh/XilunWu/175/base 2025-12-04T09:23:39.7796268Z * [new branch] gh/XilunWu/175/head -> origin/gh/XilunWu/175/head 2025-12-04T09:23:39.7796743Z * [new branch] gh/XilunWu/175/orig -> origin/gh/XilunWu/175/orig 2025-12-04T09:23:39.7797089Z * [new branch] gh/XilunWu/176/base -> origin/gh/XilunWu/176/base 2025-12-04T09:23:39.7797441Z * [new branch] gh/XilunWu/176/head -> origin/gh/XilunWu/176/head 2025-12-04T09:23:39.7797827Z * [new branch] gh/XilunWu/176/orig -> origin/gh/XilunWu/176/orig 2025-12-04T09:23:39.7798192Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-12-04T09:23:39.7798566Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-12-04T09:23:39.7798930Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-12-04T09:23:39.7799284Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-12-04T09:23:39.7799655Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-12-04T09:23:39.7800019Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-12-04T09:23:39.7800593Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-12-04T09:23:39.7800981Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-12-04T09:23:39.7801349Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-12-04T09:23:39.7801727Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-12-04T09:23:39.7802106Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-12-04T09:23:39.7802472Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-12-04T09:23:39.7802843Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-12-04T09:23:39.7803225Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-12-04T09:23:39.7803589Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-12-04T09:23:39.7803953Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-12-04T09:23:39.7804814Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-12-04T09:23:39.7805258Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-12-04T09:23:39.7805875Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-12-04T09:23:39.7806537Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-12-04T09:23:39.7807266Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-12-04T09:23:39.7808425Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-12-04T09:23:39.7808941Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-12-04T09:23:39.7809593Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-12-04T09:23:39.7810813Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-12-04T09:23:39.7812420Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-12-04T09:23:39.7812956Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-12-04T09:23:39.7813316Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-12-04T09:23:39.7813714Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-12-04T09:23:39.7814217Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-12-04T09:23:39.7818792Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-12-04T09:23:39.7819232Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-12-04T09:23:39.7819580Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-12-04T09:23:39.7820101Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-12-04T09:23:39.7820467Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-12-04T09:23:39.7820821Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-12-04T09:23:39.7821174Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-12-04T09:23:39.7821567Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-12-04T09:23:39.7821922Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-12-04T09:23:39.7822463Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-12-04T09:23:39.7822962Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-12-04T09:23:39.7823509Z * [new branch] gh/XuehaiPan/390/base -> origin/gh/XuehaiPan/390/base 2025-12-04T09:23:39.7824190Z * [new branch] gh/XuehaiPan/390/head -> origin/gh/XuehaiPan/390/head 2025-12-04T09:23:39.7825090Z * [new branch] gh/XuehaiPan/390/orig -> origin/gh/XuehaiPan/390/orig 2025-12-04T09:23:39.7826012Z * [new branch] gh/XuehaiPan/391/base -> origin/gh/XuehaiPan/391/base 2025-12-04T09:23:39.7826658Z * [new branch] gh/XuehaiPan/391/head -> origin/gh/XuehaiPan/391/head 2025-12-04T09:23:39.7827379Z * [new branch] gh/XuehaiPan/391/orig -> origin/gh/XuehaiPan/391/orig 2025-12-04T09:23:39.7828383Z * [new branch] gh/XuehaiPan/392/base -> origin/gh/XuehaiPan/392/base 2025-12-04T09:23:39.7829045Z * [new branch] gh/XuehaiPan/392/head -> origin/gh/XuehaiPan/392/head 2025-12-04T09:23:39.7829684Z * [new branch] gh/XuehaiPan/392/orig -> origin/gh/XuehaiPan/392/orig 2025-12-04T09:23:39.7837714Z * [new branch] gh/XuehaiPan/394/base -> origin/gh/XuehaiPan/394/base 2025-12-04T09:23:39.7838161Z * [new branch] gh/XuehaiPan/394/head -> origin/gh/XuehaiPan/394/head 2025-12-04T09:23:39.7838555Z * [new branch] gh/XuehaiPan/394/orig -> origin/gh/XuehaiPan/394/orig 2025-12-04T09:23:39.7838912Z * [new branch] gh/XuehaiPan/397/base -> origin/gh/XuehaiPan/397/base 2025-12-04T09:23:39.7839277Z * [new branch] gh/XuehaiPan/397/head -> origin/gh/XuehaiPan/397/head 2025-12-04T09:23:39.7839637Z * [new branch] gh/XuehaiPan/397/orig -> origin/gh/XuehaiPan/397/orig 2025-12-04T09:23:39.7840012Z * [new branch] gh/XuehaiPan/398/base -> origin/gh/XuehaiPan/398/base 2025-12-04T09:23:39.7840562Z * [new branch] gh/XuehaiPan/398/head -> origin/gh/XuehaiPan/398/head 2025-12-04T09:23:39.7840940Z * [new branch] gh/XuehaiPan/398/orig -> origin/gh/XuehaiPan/398/orig 2025-12-04T09:23:39.7841311Z * [new branch] gh/XuehaiPan/399/base -> origin/gh/XuehaiPan/399/base 2025-12-04T09:23:39.7841678Z * [new branch] gh/XuehaiPan/399/head -> origin/gh/XuehaiPan/399/head 2025-12-04T09:23:39.7842035Z * [new branch] gh/XuehaiPan/399/orig -> origin/gh/XuehaiPan/399/orig 2025-12-04T09:23:39.7842400Z * [new branch] gh/XuehaiPan/400/base -> origin/gh/XuehaiPan/400/base 2025-12-04T09:23:39.7844922Z * [new branch] gh/XuehaiPan/400/head -> origin/gh/XuehaiPan/400/head 2025-12-04T09:23:39.7845291Z * [new branch] gh/XuehaiPan/400/orig -> origin/gh/XuehaiPan/400/orig 2025-12-04T09:23:39.7846025Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-12-04T09:23:39.7846415Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-12-04T09:23:39.7846797Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-12-04T09:23:39.7851875Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-12-04T09:23:39.7856884Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-12-04T09:23:39.7861243Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-12-04T09:23:39.7865661Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-12-04T09:23:39.7869553Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-12-04T09:23:39.7874548Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-12-04T09:23:39.7876959Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-12-04T09:23:39.7877425Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-12-04T09:23:39.7877823Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-12-04T09:23:39.7878223Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-12-04T09:23:39.7878605Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-12-04T09:23:39.7878987Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-12-04T09:23:39.7879370Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-12-04T09:23:39.7879750Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-12-04T09:23:39.7880128Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-12-04T09:23:39.7880771Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-12-04T09:23:39.7881150Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-12-04T09:23:39.7881506Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-12-04T09:23:39.7881863Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-12-04T09:23:39.7882234Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-12-04T09:23:39.7882605Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-12-04T09:23:39.7882954Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-12-04T09:23:39.7883312Z * [new branch] gh/albanD/4/base -> origin/gh/albanD/4/base 2025-12-04T09:23:39.7883664Z * [new branch] gh/albanD/4/head -> origin/gh/albanD/4/head 2025-12-04T09:23:39.7884003Z * [new branch] gh/albanD/4/orig -> origin/gh/albanD/4/orig 2025-12-04T09:23:39.7884507Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-12-04T09:23:39.7885023Z * [new branch] gh/alexsamardzic/12/base -> origin/gh/alexsamardzic/12/base 2025-12-04T09:23:39.7885422Z * [new branch] gh/alexsamardzic/12/head -> origin/gh/alexsamardzic/12/head 2025-12-04T09:23:39.7885814Z * [new branch] gh/alexsamardzic/12/orig -> origin/gh/alexsamardzic/12/orig 2025-12-04T09:23:39.7886190Z * [new branch] gh/alexsamardzic/14/base -> origin/gh/alexsamardzic/14/base 2025-12-04T09:23:39.7886570Z * [new branch] gh/alexsamardzic/14/head -> origin/gh/alexsamardzic/14/head 2025-12-04T09:23:39.7887158Z * [new branch] gh/alexsamardzic/14/orig -> origin/gh/alexsamardzic/14/orig 2025-12-04T09:23:39.7887538Z * [new branch] gh/alexsamardzic/15/base -> origin/gh/alexsamardzic/15/base 2025-12-04T09:23:39.7887935Z * [new branch] gh/alexsamardzic/15/head -> origin/gh/alexsamardzic/15/head 2025-12-04T09:23:39.7888390Z * [new branch] gh/alexsamardzic/15/orig -> origin/gh/alexsamardzic/15/orig 2025-12-04T09:23:39.7888761Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-12-04T09:23:39.7889119Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-12-04T09:23:39.7889466Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-12-04T09:23:39.7889824Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-12-04T09:23:39.7890191Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-12-04T09:23:39.7890550Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-12-04T09:23:39.7890908Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-12-04T09:23:39.7891264Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-12-04T09:23:39.7891628Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-12-04T09:23:39.7891987Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-12-04T09:23:39.7892338Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-12-04T09:23:39.7892695Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-12-04T09:23:39.7893055Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-12-04T09:23:39.7894031Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-12-04T09:23:39.7894885Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-12-04T09:23:39.7895336Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-12-04T09:23:39.7895752Z * [new branch] gh/andyanwang/42/base -> origin/gh/andyanwang/42/base 2025-12-04T09:23:39.7896143Z * [new branch] gh/andyanwang/42/head -> origin/gh/andyanwang/42/head 2025-12-04T09:23:39.7896526Z * [new branch] gh/andyanwang/42/orig -> origin/gh/andyanwang/42/orig 2025-12-04T09:23:39.7896903Z * [new branch] gh/andyanwang/45/base -> origin/gh/andyanwang/45/base 2025-12-04T09:23:39.7897305Z * [new branch] gh/andyanwang/45/head -> origin/gh/andyanwang/45/head 2025-12-04T09:23:39.7897684Z * [new branch] gh/andyanwang/45/orig -> origin/gh/andyanwang/45/orig 2025-12-04T09:23:39.7898079Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-12-04T09:23:39.7898461Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-12-04T09:23:39.7898829Z * [new branch] gh/angelayi/114/base -> origin/gh/angelayi/114/base 2025-12-04T09:23:39.7899225Z * [new branch] gh/angelayi/114/head -> origin/gh/angelayi/114/head 2025-12-04T09:23:39.7900082Z * [new branch] gh/angelayi/114/orig -> origin/gh/angelayi/114/orig 2025-12-04T09:23:39.7901099Z * [new branch] gh/angelayi/116/base -> origin/gh/angelayi/116/base 2025-12-04T09:23:39.7901650Z * [new branch] gh/angelayi/116/head -> origin/gh/angelayi/116/head 2025-12-04T09:23:39.7902423Z * [new branch] gh/angelayi/116/orig -> origin/gh/angelayi/116/orig 2025-12-04T09:23:39.7904011Z * [new branch] gh/angelayi/122/base -> origin/gh/angelayi/122/base 2025-12-04T09:23:39.7905150Z * [new branch] gh/angelayi/122/head -> origin/gh/angelayi/122/head 2025-12-04T09:23:39.7905554Z * [new branch] gh/angelayi/122/orig -> origin/gh/angelayi/122/orig 2025-12-04T09:23:39.7905957Z * [new branch] gh/angelayi/124/base -> origin/gh/angelayi/124/base 2025-12-04T09:23:39.7906644Z * [new branch] gh/angelayi/124/head -> origin/gh/angelayi/124/head 2025-12-04T09:23:39.7907323Z * [new branch] gh/angelayi/124/orig -> origin/gh/angelayi/124/orig 2025-12-04T09:23:39.7908588Z * [new branch] gh/angelayi/128/base -> origin/gh/angelayi/128/base 2025-12-04T09:23:39.7909067Z * [new branch] gh/angelayi/128/head -> origin/gh/angelayi/128/head 2025-12-04T09:23:39.7909837Z * [new branch] gh/angelayi/128/orig -> origin/gh/angelayi/128/orig 2025-12-04T09:23:39.7911058Z * [new branch] gh/angelayi/131/base -> origin/gh/angelayi/131/base 2025-12-04T09:23:39.7911437Z * [new branch] gh/angelayi/131/head -> origin/gh/angelayi/131/head 2025-12-04T09:23:39.7912211Z * [new branch] gh/angelayi/131/orig -> origin/gh/angelayi/131/orig 2025-12-04T09:23:39.7913605Z * [new branch] gh/angelayi/132/base -> origin/gh/angelayi/132/base 2025-12-04T09:23:39.7914502Z * [new branch] gh/angelayi/132/head -> origin/gh/angelayi/132/head 2025-12-04T09:23:39.7915226Z * [new branch] gh/angelayi/132/orig -> origin/gh/angelayi/132/orig 2025-12-04T09:23:39.7916476Z * [new branch] gh/angelayi/133/base -> origin/gh/angelayi/133/base 2025-12-04T09:23:39.7916982Z * [new branch] gh/angelayi/133/head -> origin/gh/angelayi/133/head 2025-12-04T09:23:39.7917617Z * [new branch] gh/angelayi/133/orig -> origin/gh/angelayi/133/orig 2025-12-04T09:23:39.7919171Z * [new branch] gh/angelayi/134/base -> origin/gh/angelayi/134/base 2025-12-04T09:23:39.7919704Z * [new branch] gh/angelayi/134/head -> origin/gh/angelayi/134/head 2025-12-04T09:23:39.7920646Z * [new branch] gh/angelayi/134/orig -> origin/gh/angelayi/134/orig 2025-12-04T09:23:39.7921914Z * [new branch] gh/angelayi/135/base -> origin/gh/angelayi/135/base 2025-12-04T09:23:39.7924207Z * [new branch] gh/angelayi/135/head -> origin/gh/angelayi/135/head 2025-12-04T09:23:39.7924665Z * [new branch] gh/angelayi/135/orig -> origin/gh/angelayi/135/orig 2025-12-04T09:23:39.7925053Z * [new branch] gh/angelayi/136/base -> origin/gh/angelayi/136/base 2025-12-04T09:23:39.7925429Z * [new branch] gh/angelayi/136/head -> origin/gh/angelayi/136/head 2025-12-04T09:23:39.7925810Z * [new branch] gh/angelayi/136/orig -> origin/gh/angelayi/136/orig 2025-12-04T09:23:39.7926597Z * [new branch] gh/angelayi/137/base -> origin/gh/angelayi/137/base 2025-12-04T09:23:39.7927254Z * [new branch] gh/angelayi/137/head -> origin/gh/angelayi/137/head 2025-12-04T09:23:39.7928282Z * [new branch] gh/angelayi/137/orig -> origin/gh/angelayi/137/orig 2025-12-04T09:23:39.7929255Z * [new branch] gh/angelayi/138/base -> origin/gh/angelayi/138/base 2025-12-04T09:23:39.7929821Z * [new branch] gh/angelayi/138/head -> origin/gh/angelayi/138/head 2025-12-04T09:23:39.7931474Z * [new branch] gh/angelayi/138/orig -> origin/gh/angelayi/138/orig 2025-12-04T09:23:39.7931851Z * [new branch] gh/angelayi/139/base -> origin/gh/angelayi/139/base 2025-12-04T09:23:39.7932297Z * [new branch] gh/angelayi/139/head -> origin/gh/angelayi/139/head 2025-12-04T09:23:39.7932856Z * [new branch] gh/angelayi/139/orig -> origin/gh/angelayi/139/orig 2025-12-04T09:23:39.7937095Z * [new branch] gh/angelayi/140/base -> origin/gh/angelayi/140/base 2025-12-04T09:23:39.7940664Z * [new branch] gh/angelayi/140/head -> origin/gh/angelayi/140/head 2025-12-04T09:23:39.7941052Z * [new branch] gh/angelayi/140/orig -> origin/gh/angelayi/140/orig 2025-12-04T09:23:39.7941609Z * [new branch] gh/angelayi/141/base -> origin/gh/angelayi/141/base 2025-12-04T09:23:39.7942043Z * [new branch] gh/angelayi/141/head -> origin/gh/angelayi/141/head 2025-12-04T09:23:39.7942423Z * [new branch] gh/angelayi/141/orig -> origin/gh/angelayi/141/orig 2025-12-04T09:23:39.7942790Z * [new branch] gh/angelayi/142/base -> origin/gh/angelayi/142/base 2025-12-04T09:23:39.7943150Z * [new branch] gh/angelayi/142/head -> origin/gh/angelayi/142/head 2025-12-04T09:23:39.7943517Z * [new branch] gh/angelayi/142/orig -> origin/gh/angelayi/142/orig 2025-12-04T09:23:39.7943894Z * [new branch] gh/angelayi/143/base -> origin/gh/angelayi/143/base 2025-12-04T09:23:39.7944257Z * [new branch] gh/angelayi/143/head -> origin/gh/angelayi/143/head 2025-12-04T09:23:39.7944618Z * [new branch] gh/angelayi/143/orig -> origin/gh/angelayi/143/orig 2025-12-04T09:23:39.7945132Z * [new branch] gh/angelayi/144/base -> origin/gh/angelayi/144/base 2025-12-04T09:23:39.7945568Z * [new branch] gh/angelayi/144/head -> origin/gh/angelayi/144/head 2025-12-04T09:23:39.7946019Z * [new branch] gh/angelayi/144/orig -> origin/gh/angelayi/144/orig 2025-12-04T09:23:39.7947807Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-12-04T09:23:39.7953036Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-12-04T09:23:39.7953483Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-12-04T09:23:39.7953905Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-12-04T09:23:39.7954301Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-12-04T09:23:39.7954677Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-12-04T09:23:39.7955246Z * [new branch] gh/anijain2305/854/base -> origin/gh/anijain2305/854/base 2025-12-04T09:23:39.7955625Z * [new branch] gh/anijain2305/854/head -> origin/gh/anijain2305/854/head 2025-12-04T09:23:39.7956376Z * [new branch] gh/anijain2305/854/orig -> origin/gh/anijain2305/854/orig 2025-12-04T09:23:39.7957696Z * [new branch] gh/anijain2305/864/base -> origin/gh/anijain2305/864/base 2025-12-04T09:23:39.7958457Z * [new branch] gh/anijain2305/864/head -> origin/gh/anijain2305/864/head 2025-12-04T09:23:39.7959180Z * [new branch] gh/anijain2305/864/orig -> origin/gh/anijain2305/864/orig 2025-12-04T09:23:39.7959937Z * [new branch] gh/anijain2305/870/base -> origin/gh/anijain2305/870/base 2025-12-04T09:23:39.7960770Z * [new branch] gh/anijain2305/870/head -> origin/gh/anijain2305/870/head 2025-12-04T09:23:39.7961406Z * [new branch] gh/anijain2305/870/orig -> origin/gh/anijain2305/870/orig 2025-12-04T09:23:39.7964058Z * [new branch] gh/anijain2305/873/base -> origin/gh/anijain2305/873/base 2025-12-04T09:23:39.7964666Z * [new branch] gh/anijain2305/873/head -> origin/gh/anijain2305/873/head 2025-12-04T09:23:39.7965190Z * [new branch] gh/anijain2305/873/orig -> origin/gh/anijain2305/873/orig 2025-12-04T09:23:39.7965582Z * [new branch] gh/anijain2305/894/base -> origin/gh/anijain2305/894/base 2025-12-04T09:23:39.7966181Z * [new branch] gh/anijain2305/894/head -> origin/gh/anijain2305/894/head 2025-12-04T09:23:39.7966917Z * [new branch] gh/anijain2305/894/orig -> origin/gh/anijain2305/894/orig 2025-12-04T09:23:39.7967340Z * [new branch] gh/anijain2305/895/base -> origin/gh/anijain2305/895/base 2025-12-04T09:23:39.7968581Z * [new branch] gh/anijain2305/895/head -> origin/gh/anijain2305/895/head 2025-12-04T09:23:39.7969276Z * [new branch] gh/anijain2305/895/orig -> origin/gh/anijain2305/895/orig 2025-12-04T09:23:39.7969840Z * [new branch] gh/anijain2305/910/base -> origin/gh/anijain2305/910/base 2025-12-04T09:23:39.7972317Z * [new branch] gh/anijain2305/910/head -> origin/gh/anijain2305/910/head 2025-12-04T09:23:39.7972783Z * [new branch] gh/anijain2305/910/orig -> origin/gh/anijain2305/910/orig 2025-12-04T09:23:39.7973186Z * [new branch] gh/anijain2305/919/base -> origin/gh/anijain2305/919/base 2025-12-04T09:23:39.7973771Z * [new branch] gh/anijain2305/919/head -> origin/gh/anijain2305/919/head 2025-12-04T09:23:39.7974786Z * [new branch] gh/anijain2305/919/orig -> origin/gh/anijain2305/919/orig 2025-12-04T09:23:39.7975286Z * [new branch] gh/anijain2305/922/base -> origin/gh/anijain2305/922/base 2025-12-04T09:23:39.7975901Z * [new branch] gh/anijain2305/922/head -> origin/gh/anijain2305/922/head 2025-12-04T09:23:39.7976530Z * [new branch] gh/anijain2305/922/orig -> origin/gh/anijain2305/922/orig 2025-12-04T09:23:39.7978677Z * [new branch] gh/anijain2305/932/base -> origin/gh/anijain2305/932/base 2025-12-04T09:23:39.7979298Z * [new branch] gh/anijain2305/932/head -> origin/gh/anijain2305/932/head 2025-12-04T09:23:39.7979846Z * [new branch] gh/anijain2305/932/orig -> origin/gh/anijain2305/932/orig 2025-12-04T09:23:39.7980372Z * [new branch] gh/anijain2305/940/base -> origin/gh/anijain2305/940/base 2025-12-04T09:23:39.7981244Z * [new branch] gh/anijain2305/940/head -> origin/gh/anijain2305/940/head 2025-12-04T09:23:39.7981709Z * [new branch] gh/anijain2305/940/orig -> origin/gh/anijain2305/940/orig 2025-12-04T09:23:39.7985636Z * [new branch] gh/anijain2305/941/base -> origin/gh/anijain2305/941/base 2025-12-04T09:23:39.7986260Z * [new branch] gh/anijain2305/941/head -> origin/gh/anijain2305/941/head 2025-12-04T09:23:39.7986811Z * [new branch] gh/anijain2305/941/orig -> origin/gh/anijain2305/941/orig 2025-12-04T09:23:39.7987192Z * [new branch] gh/anijain2305/942/base -> origin/gh/anijain2305/942/base 2025-12-04T09:23:39.7987560Z * [new branch] gh/anijain2305/942/head -> origin/gh/anijain2305/942/head 2025-12-04T09:23:39.7987929Z * [new branch] gh/anijain2305/942/orig -> origin/gh/anijain2305/942/orig 2025-12-04T09:23:39.7988314Z * [new branch] gh/anijain2305/943/base -> origin/gh/anijain2305/943/base 2025-12-04T09:23:39.7988813Z * [new branch] gh/anijain2305/943/head -> origin/gh/anijain2305/943/head 2025-12-04T09:23:39.7991658Z * [new branch] gh/anijain2305/943/orig -> origin/gh/anijain2305/943/orig 2025-12-04T09:23:39.7992284Z * [new branch] gh/anijain2305/944/base -> origin/gh/anijain2305/944/base 2025-12-04T09:23:39.7992844Z * [new branch] gh/anijain2305/944/head -> origin/gh/anijain2305/944/head 2025-12-04T09:23:39.7993245Z * [new branch] gh/anijain2305/944/orig -> origin/gh/anijain2305/944/orig 2025-12-04T09:23:39.7993643Z * [new branch] gh/anijain2305/945/base -> origin/gh/anijain2305/945/base 2025-12-04T09:23:39.7994230Z * [new branch] gh/anijain2305/945/head -> origin/gh/anijain2305/945/head 2025-12-04T09:23:39.7994978Z * [new branch] gh/anijain2305/945/orig -> origin/gh/anijain2305/945/orig 2025-12-04T09:23:39.7995964Z * [new branch] gh/anijain2305/946/base -> origin/gh/anijain2305/946/base 2025-12-04T09:23:39.7996689Z * [new branch] gh/anijain2305/946/head -> origin/gh/anijain2305/946/head 2025-12-04T09:23:39.7997456Z * [new branch] gh/anijain2305/946/orig -> origin/gh/anijain2305/946/orig 2025-12-04T09:23:39.7998718Z * [new branch] gh/anijain2305/947/base -> origin/gh/anijain2305/947/base 2025-12-04T09:23:39.7999200Z * [new branch] gh/anijain2305/947/head -> origin/gh/anijain2305/947/head 2025-12-04T09:23:39.7999968Z * [new branch] gh/anijain2305/947/orig -> origin/gh/anijain2305/947/orig 2025-12-04T09:23:39.8001312Z * [new branch] gh/anijain2305/948/base -> origin/gh/anijain2305/948/base 2025-12-04T09:23:39.8002075Z * [new branch] gh/anijain2305/948/head -> origin/gh/anijain2305/948/head 2025-12-04T09:23:39.8002556Z * [new branch] gh/anijain2305/948/orig -> origin/gh/anijain2305/948/orig 2025-12-04T09:23:39.8004638Z * [new branch] gh/anijain2305/949/base -> origin/gh/anijain2305/949/base 2025-12-04T09:23:39.8005088Z * [new branch] gh/anijain2305/949/head -> origin/gh/anijain2305/949/head 2025-12-04T09:23:39.8005487Z * [new branch] gh/anijain2305/949/orig -> origin/gh/anijain2305/949/orig 2025-12-04T09:23:39.8005889Z * [new branch] gh/anijain2305/950/base -> origin/gh/anijain2305/950/base 2025-12-04T09:23:39.8011606Z * [new branch] gh/anijain2305/950/head -> origin/gh/anijain2305/950/head 2025-12-04T09:23:39.8016281Z * [new branch] gh/anijain2305/950/orig -> origin/gh/anijain2305/950/orig 2025-12-04T09:23:39.8019972Z * [new branch] gh/anijain2305/951/base -> origin/gh/anijain2305/951/base 2025-12-04T09:23:39.8024896Z * [new branch] gh/anijain2305/951/head -> origin/gh/anijain2305/951/head 2025-12-04T09:23:39.8029706Z * [new branch] gh/anijain2305/951/orig -> origin/gh/anijain2305/951/orig 2025-12-04T09:23:39.8034642Z * [new branch] gh/anijain2305/952/base -> origin/gh/anijain2305/952/base 2025-12-04T09:23:39.8036006Z * [new branch] gh/anijain2305/952/head -> origin/gh/anijain2305/952/head 2025-12-04T09:23:39.8036423Z * [new branch] gh/anijain2305/952/orig -> origin/gh/anijain2305/952/orig 2025-12-04T09:23:39.8036841Z * [new branch] gh/anijain2305/953/base -> origin/gh/anijain2305/953/base 2025-12-04T09:23:39.8037197Z * [new branch] gh/anijain2305/953/head -> origin/gh/anijain2305/953/head 2025-12-04T09:23:39.8037604Z * [new branch] gh/anijain2305/953/orig -> origin/gh/anijain2305/953/orig 2025-12-04T09:23:39.8037971Z * [new branch] gh/anijain2305/954/base -> origin/gh/anijain2305/954/base 2025-12-04T09:23:39.8038337Z * [new branch] gh/anijain2305/954/head -> origin/gh/anijain2305/954/head 2025-12-04T09:23:39.8038698Z * [new branch] gh/anijain2305/954/orig -> origin/gh/anijain2305/954/orig 2025-12-04T09:23:39.8039060Z * [new branch] gh/anijain2305/955/base -> origin/gh/anijain2305/955/base 2025-12-04T09:23:39.8039429Z * [new branch] gh/anijain2305/955/head -> origin/gh/anijain2305/955/head 2025-12-04T09:23:39.8039786Z * [new branch] gh/anijain2305/955/orig -> origin/gh/anijain2305/955/orig 2025-12-04T09:23:39.8040148Z * [new branch] gh/anijain2305/956/base -> origin/gh/anijain2305/956/base 2025-12-04T09:23:39.8040602Z * [new branch] gh/anijain2305/956/head -> origin/gh/anijain2305/956/head 2025-12-04T09:23:39.8040970Z * [new branch] gh/anijain2305/956/orig -> origin/gh/anijain2305/956/orig 2025-12-04T09:23:39.8041343Z * [new branch] gh/anijain2305/957/base -> origin/gh/anijain2305/957/base 2025-12-04T09:23:39.8041844Z * [new branch] gh/anijain2305/957/head -> origin/gh/anijain2305/957/head 2025-12-04T09:23:39.8042206Z * [new branch] gh/anijain2305/957/orig -> origin/gh/anijain2305/957/orig 2025-12-04T09:23:39.8042572Z * [new branch] gh/anijain2305/958/base -> origin/gh/anijain2305/958/base 2025-12-04T09:23:39.8043036Z * [new branch] gh/anijain2305/958/head -> origin/gh/anijain2305/958/head 2025-12-04T09:23:39.8043396Z * [new branch] gh/anijain2305/958/orig -> origin/gh/anijain2305/958/orig 2025-12-04T09:23:39.8043751Z * [new branch] gh/anijain2305/959/base -> origin/gh/anijain2305/959/base 2025-12-04T09:23:39.8044107Z * [new branch] gh/anijain2305/959/head -> origin/gh/anijain2305/959/head 2025-12-04T09:23:39.8044460Z * [new branch] gh/anijain2305/959/orig -> origin/gh/anijain2305/959/orig 2025-12-04T09:23:39.8044821Z * [new branch] gh/anijain2305/960/base -> origin/gh/anijain2305/960/base 2025-12-04T09:23:39.8045178Z * [new branch] gh/anijain2305/960/head -> origin/gh/anijain2305/960/head 2025-12-04T09:23:39.8045539Z * [new branch] gh/anijain2305/960/orig -> origin/gh/anijain2305/960/orig 2025-12-04T09:23:39.8045895Z * [new branch] gh/anijain2305/961/base -> origin/gh/anijain2305/961/base 2025-12-04T09:23:39.8046251Z * [new branch] gh/anijain2305/961/head -> origin/gh/anijain2305/961/head 2025-12-04T09:23:39.8046608Z * [new branch] gh/anijain2305/961/orig -> origin/gh/anijain2305/961/orig 2025-12-04T09:23:39.8046966Z * [new branch] gh/anijain2305/962/base -> origin/gh/anijain2305/962/base 2025-12-04T09:23:39.8047456Z * [new branch] gh/anijain2305/962/head -> origin/gh/anijain2305/962/head 2025-12-04T09:23:39.8047822Z * [new branch] gh/anijain2305/962/orig -> origin/gh/anijain2305/962/orig 2025-12-04T09:23:39.8048192Z * [new branch] gh/anijain2305/963/base -> origin/gh/anijain2305/963/base 2025-12-04T09:23:39.8048543Z * [new branch] gh/anijain2305/963/head -> origin/gh/anijain2305/963/head 2025-12-04T09:23:39.8048906Z * [new branch] gh/anijain2305/963/orig -> origin/gh/anijain2305/963/orig 2025-12-04T09:23:39.8049269Z * [new branch] gh/anijain2305/964/base -> origin/gh/anijain2305/964/base 2025-12-04T09:23:39.8049623Z * [new branch] gh/anijain2305/964/head -> origin/gh/anijain2305/964/head 2025-12-04T09:23:39.8049954Z * [new branch] gh/anijain2305/964/orig -> origin/gh/anijain2305/964/orig 2025-12-04T09:23:39.8050293Z * [new branch] gh/anijain2305/965/base -> origin/gh/anijain2305/965/base 2025-12-04T09:23:39.8050631Z * [new branch] gh/anijain2305/965/head -> origin/gh/anijain2305/965/head 2025-12-04T09:23:39.8050975Z * [new branch] gh/anijain2305/965/orig -> origin/gh/anijain2305/965/orig 2025-12-04T09:23:39.8051313Z * [new branch] gh/anijain2305/966/base -> origin/gh/anijain2305/966/base 2025-12-04T09:23:39.8051659Z * [new branch] gh/anijain2305/966/head -> origin/gh/anijain2305/966/head 2025-12-04T09:23:39.8052008Z * [new branch] gh/anijain2305/966/orig -> origin/gh/anijain2305/966/orig 2025-12-04T09:23:39.8052353Z * [new branch] gh/anijain2305/967/base -> origin/gh/anijain2305/967/base 2025-12-04T09:23:39.8052695Z * [new branch] gh/anijain2305/967/head -> origin/gh/anijain2305/967/head 2025-12-04T09:23:39.8053041Z * [new branch] gh/anijain2305/967/orig -> origin/gh/anijain2305/967/orig 2025-12-04T09:23:39.8053388Z * [new branch] gh/anijain2305/968/base -> origin/gh/anijain2305/968/base 2025-12-04T09:23:39.8053735Z * [new branch] gh/anijain2305/968/head -> origin/gh/anijain2305/968/head 2025-12-04T09:23:39.8054176Z * [new branch] gh/anijain2305/968/orig -> origin/gh/anijain2305/968/orig 2025-12-04T09:23:39.8054530Z * [new branch] gh/anijain2305/969/base -> origin/gh/anijain2305/969/base 2025-12-04T09:23:39.8054889Z * [new branch] gh/anijain2305/969/head -> origin/gh/anijain2305/969/head 2025-12-04T09:23:39.8055280Z * [new branch] gh/anijain2305/969/orig -> origin/gh/anijain2305/969/orig 2025-12-04T09:23:39.8055618Z * [new branch] gh/anijain2305/970/base -> origin/gh/anijain2305/970/base 2025-12-04T09:23:39.8055959Z * [new branch] gh/anijain2305/970/head -> origin/gh/anijain2305/970/head 2025-12-04T09:23:39.8056300Z * [new branch] gh/anijain2305/970/orig -> origin/gh/anijain2305/970/orig 2025-12-04T09:23:39.8056975Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-12-04T09:23:39.8057322Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-12-04T09:23:39.8057668Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-12-04T09:23:39.8058026Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-12-04T09:23:39.8058374Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-12-04T09:23:39.8059532Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-12-04T09:23:39.8059962Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-12-04T09:23:39.8067438Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-12-04T09:23:39.8067873Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-12-04T09:23:39.8068237Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-12-04T09:23:39.8068609Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-12-04T09:23:39.8068958Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-12-04T09:23:39.8069318Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-12-04T09:23:39.8069696Z * [new branch] gh/anshul-si/53/base -> origin/gh/anshul-si/53/base 2025-12-04T09:23:39.8070073Z * [new branch] gh/anshul-si/53/head -> origin/gh/anshul-si/53/head 2025-12-04T09:23:39.8070437Z * [new branch] gh/anshul-si/58/base -> origin/gh/anshul-si/58/base 2025-12-04T09:23:39.8070807Z * [new branch] gh/anshul-si/58/head -> origin/gh/anshul-si/58/head 2025-12-04T09:23:39.8071174Z * [new branch] gh/anshul-si/66/base -> origin/gh/anshul-si/66/base 2025-12-04T09:23:39.8071726Z * [new branch] gh/anshul-si/66/head -> origin/gh/anshul-si/66/head 2025-12-04T09:23:39.8072245Z * [new branch] gh/anshul-si/66/orig -> origin/gh/anshul-si/66/orig 2025-12-04T09:23:39.8072748Z * [new branch] gh/anshul-si/67/base -> origin/gh/anshul-si/67/base 2025-12-04T09:23:39.8073250Z * [new branch] gh/anshul-si/67/head -> origin/gh/anshul-si/67/head 2025-12-04T09:23:39.8073765Z * [new branch] gh/anshul-si/67/orig -> origin/gh/anshul-si/67/orig 2025-12-04T09:23:39.8074159Z * [new branch] gh/anshul-si/68/base -> origin/gh/anshul-si/68/base 2025-12-04T09:23:39.8074530Z * [new branch] gh/anshul-si/68/head -> origin/gh/anshul-si/68/head 2025-12-04T09:23:39.8074955Z * [new branch] gh/anshul-si/68/orig -> origin/gh/anshul-si/68/orig 2025-12-04T09:23:39.8075314Z * [new branch] gh/anshul-si/69/base -> origin/gh/anshul-si/69/base 2025-12-04T09:23:39.8075670Z * [new branch] gh/anshul-si/69/head -> origin/gh/anshul-si/69/head 2025-12-04T09:23:39.8076178Z * [new branch] gh/anshul-si/69/orig -> origin/gh/anshul-si/69/orig 2025-12-04T09:23:39.8076557Z * [new branch] gh/anshul-si/70/base -> origin/gh/anshul-si/70/base 2025-12-04T09:23:39.8076921Z * [new branch] gh/anshul-si/70/head -> origin/gh/anshul-si/70/head 2025-12-04T09:23:39.8077352Z * [new branch] gh/anshul-si/70/orig -> origin/gh/anshul-si/70/orig 2025-12-04T09:23:39.8078406Z * [new branch] gh/anshul-si/71/base -> origin/gh/anshul-si/71/base 2025-12-04T09:23:39.8079099Z * [new branch] gh/anshul-si/71/head -> origin/gh/anshul-si/71/head 2025-12-04T09:23:39.8079816Z * [new branch] gh/anshul-si/71/orig -> origin/gh/anshul-si/71/orig 2025-12-04T09:23:39.8081651Z * [new branch] gh/anshul-si/72/base -> origin/gh/anshul-si/72/base 2025-12-04T09:23:39.8082164Z * [new branch] gh/anshul-si/72/head -> origin/gh/anshul-si/72/head 2025-12-04T09:23:39.8082666Z * [new branch] gh/anshul-si/72/orig -> origin/gh/anshul-si/72/orig 2025-12-04T09:23:39.8083448Z * [new branch] gh/anshul-si/73/base -> origin/gh/anshul-si/73/base 2025-12-04T09:23:39.8084143Z * [new branch] gh/anshul-si/73/head -> origin/gh/anshul-si/73/head 2025-12-04T09:23:39.8084772Z * [new branch] gh/anshul-si/73/orig -> origin/gh/anshul-si/73/orig 2025-12-04T09:23:39.8088887Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-12-04T09:23:39.8089497Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-12-04T09:23:39.8089950Z * [new branch] gh/aorenste/134/base -> origin/gh/aorenste/134/base 2025-12-04T09:23:39.8090828Z * [new branch] gh/aorenste/134/head -> origin/gh/aorenste/134/head 2025-12-04T09:23:39.8091281Z * [new branch] gh/aorenste/134/orig -> origin/gh/aorenste/134/orig 2025-12-04T09:23:39.8091657Z * [new branch] gh/aorenste/139/base -> origin/gh/aorenste/139/base 2025-12-04T09:23:39.8092021Z * [new branch] gh/aorenste/139/head -> origin/gh/aorenste/139/head 2025-12-04T09:23:39.8092396Z * [new branch] gh/aorenste/139/orig -> origin/gh/aorenste/139/orig 2025-12-04T09:23:39.8093025Z * [new branch] gh/aorenste/141/base -> origin/gh/aorenste/141/base 2025-12-04T09:23:39.8093573Z * [new branch] gh/aorenste/141/head -> origin/gh/aorenste/141/head 2025-12-04T09:23:39.8095978Z * [new branch] gh/aorenste/145/base -> origin/gh/aorenste/145/base 2025-12-04T09:23:39.8096526Z * [new branch] gh/aorenste/145/head -> origin/gh/aorenste/145/head 2025-12-04T09:23:39.8097040Z * [new branch] gh/aorenste/145/orig -> origin/gh/aorenste/145/orig 2025-12-04T09:23:39.8097614Z * [new branch] gh/aorenste/146/base -> origin/gh/aorenste/146/base 2025-12-04T09:23:39.8100592Z * [new branch] gh/aorenste/146/head -> origin/gh/aorenste/146/head 2025-12-04T09:23:39.8101097Z * [new branch] gh/aorenste/146/orig -> origin/gh/aorenste/146/orig 2025-12-04T09:23:39.8101564Z * [new branch] gh/aorenste/147/base -> origin/gh/aorenste/147/base 2025-12-04T09:23:39.8102221Z * [new branch] gh/aorenste/147/head -> origin/gh/aorenste/147/head 2025-12-04T09:23:39.8103097Z * [new branch] gh/aorenste/147/orig -> origin/gh/aorenste/147/orig 2025-12-04T09:23:39.8103686Z * [new branch] gh/aorenste/148/base -> origin/gh/aorenste/148/base 2025-12-04T09:23:39.8104144Z * [new branch] gh/aorenste/148/head -> origin/gh/aorenste/148/head 2025-12-04T09:23:39.8104974Z * [new branch] gh/aorenste/148/orig -> origin/gh/aorenste/148/orig 2025-12-04T09:23:39.8105574Z * [new branch] gh/aorenste/149/base -> origin/gh/aorenste/149/base 2025-12-04T09:23:39.8106170Z * [new branch] gh/aorenste/149/head -> origin/gh/aorenste/149/head 2025-12-04T09:23:39.8106675Z * [new branch] gh/aorenste/149/orig -> origin/gh/aorenste/149/orig 2025-12-04T09:23:39.8111454Z * [new branch] gh/aorenste/150/base -> origin/gh/aorenste/150/base 2025-12-04T09:23:39.8111907Z * [new branch] gh/aorenste/150/head -> origin/gh/aorenste/150/head 2025-12-04T09:23:39.8112264Z * [new branch] gh/aorenste/150/orig -> origin/gh/aorenste/150/orig 2025-12-04T09:23:39.8112622Z * [new branch] gh/aorenste/151/base -> origin/gh/aorenste/151/base 2025-12-04T09:23:39.8112974Z * [new branch] gh/aorenste/151/head -> origin/gh/aorenste/151/head 2025-12-04T09:23:39.8113316Z * [new branch] gh/aorenste/151/orig -> origin/gh/aorenste/151/orig 2025-12-04T09:23:39.8113684Z * [new branch] gh/aorenste/152/base -> origin/gh/aorenste/152/base 2025-12-04T09:23:39.8114027Z * [new branch] gh/aorenste/152/head -> origin/gh/aorenste/152/head 2025-12-04T09:23:39.8114372Z * [new branch] gh/aorenste/152/orig -> origin/gh/aorenste/152/orig 2025-12-04T09:23:39.8114725Z * [new branch] gh/aorenste/153/base -> origin/gh/aorenste/153/base 2025-12-04T09:23:39.8115146Z * [new branch] gh/aorenste/153/head -> origin/gh/aorenste/153/head 2025-12-04T09:23:39.8115822Z * [new branch] gh/aorenste/153/orig -> origin/gh/aorenste/153/orig 2025-12-04T09:23:39.8117624Z * [new branch] gh/aorenste/154/base -> origin/gh/aorenste/154/base 2025-12-04T09:23:39.8118085Z * [new branch] gh/aorenste/154/head -> origin/gh/aorenste/154/head 2025-12-04T09:23:39.8118483Z * [new branch] gh/aorenste/154/orig -> origin/gh/aorenste/154/orig 2025-12-04T09:23:39.8119058Z * [new branch] gh/aorenste/155/base -> origin/gh/aorenste/155/base 2025-12-04T09:23:39.8119558Z * [new branch] gh/aorenste/155/head -> origin/gh/aorenste/155/head 2025-12-04T09:23:39.8120270Z * [new branch] gh/aorenste/155/orig -> origin/gh/aorenste/155/orig 2025-12-04T09:23:39.8122315Z * [new branch] gh/aorenste/156/base -> origin/gh/aorenste/156/base 2025-12-04T09:23:39.8122814Z * [new branch] gh/aorenste/156/head -> origin/gh/aorenste/156/head 2025-12-04T09:23:39.8123313Z * [new branch] gh/aorenste/156/orig -> origin/gh/aorenste/156/orig 2025-12-04T09:23:39.8123777Z * [new branch] gh/aorenste/157/base -> origin/gh/aorenste/157/base 2025-12-04T09:23:39.8124531Z * [new branch] gh/aorenste/157/head -> origin/gh/aorenste/157/head 2025-12-04T09:23:39.8125150Z * [new branch] gh/aorenste/157/orig -> origin/gh/aorenste/157/orig 2025-12-04T09:23:39.8128334Z * [new branch] gh/aorenste/158/base -> origin/gh/aorenste/158/base 2025-12-04T09:23:39.8128764Z * [new branch] gh/aorenste/158/head -> origin/gh/aorenste/158/head 2025-12-04T09:23:39.8129169Z * [new branch] gh/aorenste/158/orig -> origin/gh/aorenste/158/orig 2025-12-04T09:23:39.8129537Z * [new branch] gh/aorenste/159/base -> origin/gh/aorenste/159/base 2025-12-04T09:23:39.8130078Z * [new branch] gh/aorenste/159/head -> origin/gh/aorenste/159/head 2025-12-04T09:23:39.8130571Z * [new branch] gh/aorenste/159/orig -> origin/gh/aorenste/159/orig 2025-12-04T09:23:39.8131095Z * [new branch] gh/avikchaudhuri/1/base -> origin/gh/avikchaudhuri/1/base 2025-12-04T09:23:39.8132812Z * [new branch] gh/avikchaudhuri/1/head -> origin/gh/avikchaudhuri/1/head 2025-12-04T09:23:39.8133564Z * [new branch] gh/avikchaudhuri/2/base -> origin/gh/avikchaudhuri/2/base 2025-12-04T09:23:39.8133967Z * [new branch] gh/avikchaudhuri/2/head -> origin/gh/avikchaudhuri/2/head 2025-12-04T09:23:39.8134470Z * [new branch] gh/avikchaudhuri/2/orig -> origin/gh/avikchaudhuri/2/orig 2025-12-04T09:23:39.8139155Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-12-04T09:23:39.8139765Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-12-04T09:23:39.8140733Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-12-04T09:23:39.8141179Z * [new branch] gh/bdhirsh/668/base -> origin/gh/bdhirsh/668/base 2025-12-04T09:23:39.8141551Z * [new branch] gh/bdhirsh/668/head -> origin/gh/bdhirsh/668/head 2025-12-04T09:23:39.8141978Z * [new branch] gh/bdhirsh/668/orig -> origin/gh/bdhirsh/668/orig 2025-12-04T09:23:39.8142360Z * [new branch] gh/bdhirsh/669/base -> origin/gh/bdhirsh/669/base 2025-12-04T09:23:39.8142733Z * [new branch] gh/bdhirsh/669/head -> origin/gh/bdhirsh/669/head 2025-12-04T09:23:39.8143095Z * [new branch] gh/bdhirsh/669/orig -> origin/gh/bdhirsh/669/orig 2025-12-04T09:23:39.8148576Z * [new branch] gh/bdhirsh/670/base -> origin/gh/bdhirsh/670/base 2025-12-04T09:23:39.8149015Z * [new branch] gh/bdhirsh/670/head -> origin/gh/bdhirsh/670/head 2025-12-04T09:23:39.8149415Z * [new branch] gh/bdhirsh/670/orig -> origin/gh/bdhirsh/670/orig 2025-12-04T09:23:39.8149777Z * [new branch] gh/bdhirsh/672/base -> origin/gh/bdhirsh/672/base 2025-12-04T09:23:39.8150144Z * [new branch] gh/bdhirsh/672/head -> origin/gh/bdhirsh/672/head 2025-12-04T09:23:39.8150510Z * [new branch] gh/bdhirsh/672/orig -> origin/gh/bdhirsh/672/orig 2025-12-04T09:23:39.8150867Z * [new branch] gh/bdhirsh/675/base -> origin/gh/bdhirsh/675/base 2025-12-04T09:23:39.8151221Z * [new branch] gh/bdhirsh/675/head -> origin/gh/bdhirsh/675/head 2025-12-04T09:23:39.8151573Z * [new branch] gh/bdhirsh/675/orig -> origin/gh/bdhirsh/675/orig 2025-12-04T09:23:39.8151923Z * [new branch] gh/bdhirsh/676/base -> origin/gh/bdhirsh/676/base 2025-12-04T09:23:39.8152444Z * [new branch] gh/bdhirsh/676/head -> origin/gh/bdhirsh/676/head 2025-12-04T09:23:39.8152954Z * [new branch] gh/bdhirsh/676/orig -> origin/gh/bdhirsh/676/orig 2025-12-04T09:23:39.8154766Z * [new branch] gh/bdhirsh/677/base -> origin/gh/bdhirsh/677/base 2025-12-04T09:23:39.8154950Z * [new branch] gh/bdhirsh/677/head -> origin/gh/bdhirsh/677/head 2025-12-04T09:23:39.8155128Z * [new branch] gh/bdhirsh/677/orig -> origin/gh/bdhirsh/677/orig 2025-12-04T09:23:39.8157482Z * [new branch] gh/bdhirsh/678/base -> origin/gh/bdhirsh/678/base 2025-12-04T09:23:39.8157674Z * [new branch] gh/bdhirsh/678/head -> origin/gh/bdhirsh/678/head 2025-12-04T09:23:39.8157860Z * [new branch] gh/bdhirsh/678/orig -> origin/gh/bdhirsh/678/orig 2025-12-04T09:23:39.8159011Z * [new branch] gh/bdhirsh/679/base -> origin/gh/bdhirsh/679/base 2025-12-04T09:23:39.8159273Z * [new branch] gh/bdhirsh/679/head -> origin/gh/bdhirsh/679/head 2025-12-04T09:23:39.8160511Z * [new branch] gh/bdhirsh/679/orig -> origin/gh/bdhirsh/679/orig 2025-12-04T09:23:39.8168295Z * [new branch] gh/bdhirsh/680/base -> origin/gh/bdhirsh/680/base 2025-12-04T09:23:39.8168635Z * [new branch] gh/bdhirsh/680/head -> origin/gh/bdhirsh/680/head 2025-12-04T09:23:39.8169169Z * [new branch] gh/bdhirsh/680/orig -> origin/gh/bdhirsh/680/orig 2025-12-04T09:23:39.8169475Z * [new branch] gh/bdhirsh/681/base -> origin/gh/bdhirsh/681/base 2025-12-04T09:23:39.8169657Z * [new branch] gh/bdhirsh/681/head -> origin/gh/bdhirsh/681/head 2025-12-04T09:23:39.8169952Z * [new branch] gh/bdhirsh/681/orig -> origin/gh/bdhirsh/681/orig 2025-12-04T09:23:39.8170151Z * [new branch] gh/benjaminglass1/101/base -> origin/gh/benjaminglass1/101/base 2025-12-04T09:23:39.8170458Z * [new branch] gh/benjaminglass1/101/head -> origin/gh/benjaminglass1/101/head 2025-12-04T09:23:39.8170635Z * [new branch] gh/benjaminglass1/101/orig -> origin/gh/benjaminglass1/101/orig 2025-12-04T09:23:39.8170816Z * [new branch] gh/benjaminglass1/102/base -> origin/gh/benjaminglass1/102/base 2025-12-04T09:23:39.8170990Z * [new branch] gh/benjaminglass1/102/head -> origin/gh/benjaminglass1/102/head 2025-12-04T09:23:39.8171165Z * [new branch] gh/benjaminglass1/102/orig -> origin/gh/benjaminglass1/102/orig 2025-12-04T09:23:39.8171330Z * [new branch] gh/benjaminglass1/106/base -> origin/gh/benjaminglass1/106/base 2025-12-04T09:23:39.8171512Z * [new branch] gh/benjaminglass1/106/head -> origin/gh/benjaminglass1/106/head 2025-12-04T09:23:39.8171683Z * [new branch] gh/benjaminglass1/106/orig -> origin/gh/benjaminglass1/106/orig 2025-12-04T09:23:39.8178967Z * [new branch] gh/benjaminglass1/107/base -> origin/gh/benjaminglass1/107/base 2025-12-04T09:23:39.8179358Z * [new branch] gh/benjaminglass1/107/head -> origin/gh/benjaminglass1/107/head 2025-12-04T09:23:39.8179570Z * [new branch] gh/benjaminglass1/107/orig -> origin/gh/benjaminglass1/107/orig 2025-12-04T09:23:39.8179751Z * [new branch] gh/benjaminglass1/108/base -> origin/gh/benjaminglass1/108/base 2025-12-04T09:23:39.8179940Z * [new branch] gh/benjaminglass1/108/head -> origin/gh/benjaminglass1/108/head 2025-12-04T09:23:39.8180114Z * [new branch] gh/benjaminglass1/108/orig -> origin/gh/benjaminglass1/108/orig 2025-12-04T09:23:39.8180418Z * [new branch] gh/benjaminglass1/109/base -> origin/gh/benjaminglass1/109/base 2025-12-04T09:23:39.8180741Z * [new branch] gh/benjaminglass1/109/head -> origin/gh/benjaminglass1/109/head 2025-12-04T09:23:39.8180911Z * [new branch] gh/benjaminglass1/109/orig -> origin/gh/benjaminglass1/109/orig 2025-12-04T09:23:39.8181077Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-12-04T09:23:39.8181248Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-12-04T09:23:39.8181544Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-12-04T09:23:39.8187655Z * [new branch] gh/bobrenjc93/570/base -> origin/gh/bobrenjc93/570/base 2025-12-04T09:23:39.8187898Z * [new branch] gh/bobrenjc93/570/head -> origin/gh/bobrenjc93/570/head 2025-12-04T09:23:39.8188061Z * [new branch] gh/bobrenjc93/570/orig -> origin/gh/bobrenjc93/570/orig 2025-12-04T09:23:39.8188226Z * [new branch] gh/bobrenjc93/604/base -> origin/gh/bobrenjc93/604/base 2025-12-04T09:23:39.8188383Z * [new branch] gh/bobrenjc93/604/head -> origin/gh/bobrenjc93/604/head 2025-12-04T09:23:39.8188674Z * [new branch] gh/bobrenjc93/604/orig -> origin/gh/bobrenjc93/604/orig 2025-12-04T09:23:39.8189197Z * [new branch] gh/bobrenjc93/638/base -> origin/gh/bobrenjc93/638/base 2025-12-04T09:23:39.8189346Z * [new branch] gh/bobrenjc93/638/head -> origin/gh/bobrenjc93/638/head 2025-12-04T09:23:39.8189492Z * [new branch] gh/bobrenjc93/638/orig -> origin/gh/bobrenjc93/638/orig 2025-12-04T09:23:39.8189783Z * [new branch] gh/bobrenjc93/653/base -> origin/gh/bobrenjc93/653/base 2025-12-04T09:23:39.8189935Z * [new branch] gh/bobrenjc93/653/head -> origin/gh/bobrenjc93/653/head 2025-12-04T09:23:39.8190086Z * [new branch] gh/bobrenjc93/653/orig -> origin/gh/bobrenjc93/653/orig 2025-12-04T09:23:39.8190288Z * [new branch] gh/bobrenjc93/654/base -> origin/gh/bobrenjc93/654/base 2025-12-04T09:23:39.8195754Z * [new branch] gh/bobrenjc93/654/head -> origin/gh/bobrenjc93/654/head 2025-12-04T09:23:39.8196125Z * [new branch] gh/bobrenjc93/654/orig -> origin/gh/bobrenjc93/654/orig 2025-12-04T09:23:39.8196321Z * [new branch] gh/bobrenjc93/657/base -> origin/gh/bobrenjc93/657/base 2025-12-04T09:23:39.8196487Z * [new branch] gh/bobrenjc93/657/head -> origin/gh/bobrenjc93/657/head 2025-12-04T09:23:39.8196658Z * [new branch] gh/bobrenjc93/657/orig -> origin/gh/bobrenjc93/657/orig 2025-12-04T09:23:39.8196820Z * [new branch] gh/bobrenjc93/672/base -> origin/gh/bobrenjc93/672/base 2025-12-04T09:23:39.8197105Z * [new branch] gh/bobrenjc93/672/head -> origin/gh/bobrenjc93/672/head 2025-12-04T09:23:39.8197305Z * [new branch] gh/bobrenjc93/672/orig -> origin/gh/bobrenjc93/672/orig 2025-12-04T09:23:39.8197474Z * [new branch] gh/bobrenjc93/679/base -> origin/gh/bobrenjc93/679/base 2025-12-04T09:23:39.8198168Z * [new branch] gh/bobrenjc93/679/head -> origin/gh/bobrenjc93/679/head 2025-12-04T09:23:39.8198358Z * [new branch] gh/bobrenjc93/679/orig -> origin/gh/bobrenjc93/679/orig 2025-12-04T09:23:39.8198531Z * [new branch] gh/bobrenjc93/680/base -> origin/gh/bobrenjc93/680/base 2025-12-04T09:23:39.8200595Z * [new branch] gh/bobrenjc93/680/head -> origin/gh/bobrenjc93/680/head 2025-12-04T09:23:39.8200877Z * [new branch] gh/bobrenjc93/680/orig -> origin/gh/bobrenjc93/680/orig 2025-12-04T09:23:39.8201050Z * [new branch] gh/bobrenjc93/681/base -> origin/gh/bobrenjc93/681/base 2025-12-04T09:23:39.8205435Z * [new branch] gh/bobrenjc93/681/head -> origin/gh/bobrenjc93/681/head 2025-12-04T09:23:39.8205647Z * [new branch] gh/bobrenjc93/681/orig -> origin/gh/bobrenjc93/681/orig 2025-12-04T09:23:39.8205809Z * [new branch] gh/bobrenjc93/682/base -> origin/gh/bobrenjc93/682/base 2025-12-04T09:23:39.8205970Z * [new branch] gh/bobrenjc93/682/head -> origin/gh/bobrenjc93/682/head 2025-12-04T09:23:39.8206123Z * [new branch] gh/bobrenjc93/682/orig -> origin/gh/bobrenjc93/682/orig 2025-12-04T09:23:39.8206426Z * [new branch] gh/bobrenjc93/683/base -> origin/gh/bobrenjc93/683/base 2025-12-04T09:23:39.8211774Z * [new branch] gh/bobrenjc93/683/head -> origin/gh/bobrenjc93/683/head 2025-12-04T09:23:39.8213951Z * [new branch] gh/bobrenjc93/683/orig -> origin/gh/bobrenjc93/683/orig 2025-12-04T09:23:39.8214528Z * [new branch] gh/bobrenjc93/684/base -> origin/gh/bobrenjc93/684/base 2025-12-04T09:23:39.8214747Z * [new branch] gh/bobrenjc93/684/head -> origin/gh/bobrenjc93/684/head 2025-12-04T09:23:39.8214922Z * [new branch] gh/bobrenjc93/684/orig -> origin/gh/bobrenjc93/684/orig 2025-12-04T09:23:39.8215078Z * [new branch] gh/bobrenjc93/685/base -> origin/gh/bobrenjc93/685/base 2025-12-04T09:23:39.8215242Z * [new branch] gh/bobrenjc93/685/head -> origin/gh/bobrenjc93/685/head 2025-12-04T09:23:39.8215398Z * [new branch] gh/bobrenjc93/685/orig -> origin/gh/bobrenjc93/685/orig 2025-12-04T09:23:39.8215556Z * [new branch] gh/bobrenjc93/686/base -> origin/gh/bobrenjc93/686/base 2025-12-04T09:23:39.8215843Z * [new branch] gh/bobrenjc93/686/head -> origin/gh/bobrenjc93/686/head 2025-12-04T09:23:39.8216000Z * [new branch] gh/bobrenjc93/686/orig -> origin/gh/bobrenjc93/686/orig 2025-12-04T09:23:39.8217681Z * [new branch] gh/bobrenjc93/687/base -> origin/gh/bobrenjc93/687/base 2025-12-04T09:23:39.8218003Z * [new branch] gh/bobrenjc93/687/head -> origin/gh/bobrenjc93/687/head 2025-12-04T09:23:39.8218196Z * [new branch] gh/bobrenjc93/687/orig -> origin/gh/bobrenjc93/687/orig 2025-12-04T09:23:39.8223189Z * [new branch] gh/bobrenjc93/688/base -> origin/gh/bobrenjc93/688/base 2025-12-04T09:23:39.8228650Z * [new branch] gh/bobrenjc93/688/head -> origin/gh/bobrenjc93/688/head 2025-12-04T09:23:39.8231535Z * [new branch] gh/bobrenjc93/688/orig -> origin/gh/bobrenjc93/688/orig 2025-12-04T09:23:39.8231845Z * [new branch] gh/bobrenjc93/689/base -> origin/gh/bobrenjc93/689/base 2025-12-04T09:23:39.8236830Z * [new branch] gh/bobrenjc93/689/head -> origin/gh/bobrenjc93/689/head 2025-12-04T09:23:39.8237143Z * [new branch] gh/bobrenjc93/689/orig -> origin/gh/bobrenjc93/689/orig 2025-12-04T09:23:39.8237444Z * [new branch] gh/bobrenjc93/690/base -> origin/gh/bobrenjc93/690/base 2025-12-04T09:23:39.8237622Z * [new branch] gh/bobrenjc93/690/head -> origin/gh/bobrenjc93/690/head 2025-12-04T09:23:39.8237783Z * [new branch] gh/bobrenjc93/690/orig -> origin/gh/bobrenjc93/690/orig 2025-12-04T09:23:39.8237941Z * [new branch] gh/bobrenjc93/691/base -> origin/gh/bobrenjc93/691/base 2025-12-04T09:23:39.8238098Z * [new branch] gh/bobrenjc93/691/head -> origin/gh/bobrenjc93/691/head 2025-12-04T09:23:39.8238250Z * [new branch] gh/bobrenjc93/691/orig -> origin/gh/bobrenjc93/691/orig 2025-12-04T09:23:39.8238410Z * [new branch] gh/bobrenjc93/692/base -> origin/gh/bobrenjc93/692/base 2025-12-04T09:23:39.8238567Z * [new branch] gh/bobrenjc93/692/head -> origin/gh/bobrenjc93/692/head 2025-12-04T09:23:39.8238721Z * [new branch] gh/bobrenjc93/692/orig -> origin/gh/bobrenjc93/692/orig 2025-12-04T09:23:39.8238885Z * [new branch] gh/bobrenjc93/693/base -> origin/gh/bobrenjc93/693/base 2025-12-04T09:23:39.8239035Z * [new branch] gh/bobrenjc93/693/head -> origin/gh/bobrenjc93/693/head 2025-12-04T09:23:39.8239193Z * [new branch] gh/bobrenjc93/693/orig -> origin/gh/bobrenjc93/693/orig 2025-12-04T09:23:39.8239341Z * [new branch] gh/bobrenjc93/694/base -> origin/gh/bobrenjc93/694/base 2025-12-04T09:23:39.8239494Z * [new branch] gh/bobrenjc93/694/head -> origin/gh/bobrenjc93/694/head 2025-12-04T09:23:39.8239641Z * [new branch] gh/bobrenjc93/694/orig -> origin/gh/bobrenjc93/694/orig 2025-12-04T09:23:39.8239792Z * [new branch] gh/bobrenjc93/695/base -> origin/gh/bobrenjc93/695/base 2025-12-04T09:23:39.8239949Z * [new branch] gh/bobrenjc93/695/head -> origin/gh/bobrenjc93/695/head 2025-12-04T09:23:39.8240096Z * [new branch] gh/bobrenjc93/695/orig -> origin/gh/bobrenjc93/695/orig 2025-12-04T09:23:39.8240255Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-12-04T09:23:39.8240964Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-12-04T09:23:39.8245127Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-12-04T09:23:39.8251204Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-12-04T09:23:39.8251416Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-12-04T09:23:39.8251558Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-12-04T09:23:39.8251919Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-12-04T09:23:39.8252213Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-12-04T09:23:39.8252364Z * [new branch] gh/c00w/56/base -> origin/gh/c00w/56/base 2025-12-04T09:23:39.8252575Z * [new branch] gh/c00w/56/head -> origin/gh/c00w/56/head 2025-12-04T09:23:39.8252839Z * [new branch] gh/c00w/56/orig -> origin/gh/c00w/56/orig 2025-12-04T09:23:39.8252981Z * [new branch] gh/c00w/57/base -> origin/gh/c00w/57/base 2025-12-04T09:23:39.8253303Z * [new branch] gh/c00w/57/head -> origin/gh/c00w/57/head 2025-12-04T09:23:39.8253427Z * [new branch] gh/c00w/57/orig -> origin/gh/c00w/57/orig 2025-12-04T09:23:39.8253559Z * [new branch] gh/c00w/58/base -> origin/gh/c00w/58/base 2025-12-04T09:23:39.8253694Z * [new branch] gh/c00w/58/head -> origin/gh/c00w/58/head 2025-12-04T09:23:39.8253940Z * [new branch] gh/c00w/58/orig -> origin/gh/c00w/58/orig 2025-12-04T09:23:39.8254106Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-12-04T09:23:39.8254249Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-12-04T09:23:39.8254618Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-12-04T09:23:39.8261204Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-12-04T09:23:39.8261560Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-12-04T09:23:39.8261751Z * [new branch] gh/coconutruben/55/base -> origin/gh/coconutruben/55/base 2025-12-04T09:23:39.8261996Z * [new branch] gh/coconutruben/55/head -> origin/gh/coconutruben/55/head 2025-12-04T09:23:39.8262194Z * [new branch] gh/coconutruben/55/orig -> origin/gh/coconutruben/55/orig 2025-12-04T09:23:39.8266830Z * [new branch] gh/coconutruben/57/base -> origin/gh/coconutruben/57/base 2025-12-04T09:23:39.8267184Z * [new branch] gh/coconutruben/57/head -> origin/gh/coconutruben/57/head 2025-12-04T09:23:39.8267380Z * [new branch] gh/coconutruben/57/orig -> origin/gh/coconutruben/57/orig 2025-12-04T09:23:39.8267628Z * [new branch] gh/coconutruben/70/base -> origin/gh/coconutruben/70/base 2025-12-04T09:23:39.8267943Z * [new branch] gh/coconutruben/70/head -> origin/gh/coconutruben/70/head 2025-12-04T09:23:39.8268095Z * [new branch] gh/coconutruben/70/orig -> origin/gh/coconutruben/70/orig 2025-12-04T09:23:39.8268612Z * [new branch] gh/coconutruben/71/base -> origin/gh/coconutruben/71/base 2025-12-04T09:23:39.8268805Z * [new branch] gh/coconutruben/71/head -> origin/gh/coconutruben/71/head 2025-12-04T09:23:39.8268990Z * [new branch] gh/coconutruben/71/orig -> origin/gh/coconutruben/71/orig 2025-12-04T09:23:39.8269146Z * [new branch] gh/coconutruben/72/base -> origin/gh/coconutruben/72/base 2025-12-04T09:23:39.8269318Z * [new branch] gh/coconutruben/72/head -> origin/gh/coconutruben/72/head 2025-12-04T09:23:39.8269673Z * [new branch] gh/coconutruben/72/orig -> origin/gh/coconutruben/72/orig 2025-12-04T09:23:39.8269884Z * [new branch] gh/coconutruben/73/base -> origin/gh/coconutruben/73/base 2025-12-04T09:23:39.8276285Z * [new branch] gh/coconutruben/73/head -> origin/gh/coconutruben/73/head 2025-12-04T09:23:39.8276509Z * [new branch] gh/coconutruben/73/orig -> origin/gh/coconutruben/73/orig 2025-12-04T09:23:39.8276688Z * [new branch] gh/coconutruben/74/base -> origin/gh/coconutruben/74/base 2025-12-04T09:23:39.8277026Z * [new branch] gh/coconutruben/74/head -> origin/gh/coconutruben/74/head 2025-12-04T09:23:39.8277194Z * [new branch] gh/coconutruben/74/orig -> origin/gh/coconutruben/74/orig 2025-12-04T09:23:39.8277360Z * [new branch] gh/coconutruben/79/base -> origin/gh/coconutruben/79/base 2025-12-04T09:23:39.8278017Z * [new branch] gh/coconutruben/79/head -> origin/gh/coconutruben/79/head 2025-12-04T09:23:39.8278194Z * [new branch] gh/coconutruben/79/orig -> origin/gh/coconutruben/79/orig 2025-12-04T09:23:39.8278370Z * [new branch] gh/coconutruben/80/base -> origin/gh/coconutruben/80/base 2025-12-04T09:23:39.8278540Z * [new branch] gh/coconutruben/80/head -> origin/gh/coconutruben/80/head 2025-12-04T09:23:39.8278718Z * [new branch] gh/coconutruben/80/orig -> origin/gh/coconutruben/80/orig 2025-12-04T09:23:39.8278884Z * [new branch] gh/coconutruben/82/base -> origin/gh/coconutruben/82/base 2025-12-04T09:23:39.8279406Z * [new branch] gh/coconutruben/82/head -> origin/gh/coconutruben/82/head 2025-12-04T09:23:39.8288973Z * [new branch] gh/coconutruben/82/orig -> origin/gh/coconutruben/82/orig 2025-12-04T09:23:39.8294058Z * [new branch] gh/coconutruben/83/base -> origin/gh/coconutruben/83/base 2025-12-04T09:23:39.8299002Z * [new branch] gh/coconutruben/83/head -> origin/gh/coconutruben/83/head 2025-12-04T09:23:39.8301252Z * [new branch] gh/coconutruben/83/orig -> origin/gh/coconutruben/83/orig 2025-12-04T09:23:39.8301463Z * [new branch] gh/coconutruben/84/base -> origin/gh/coconutruben/84/base 2025-12-04T09:23:39.8301707Z * [new branch] gh/coconutruben/84/head -> origin/gh/coconutruben/84/head 2025-12-04T09:23:39.8301891Z * [new branch] gh/coconutruben/84/orig -> origin/gh/coconutruben/84/orig 2025-12-04T09:23:39.8302086Z * [new branch] gh/coconutruben/85/base -> origin/gh/coconutruben/85/base 2025-12-04T09:23:39.8302263Z * [new branch] gh/coconutruben/85/head -> origin/gh/coconutruben/85/head 2025-12-04T09:23:39.8302437Z * [new branch] gh/coconutruben/85/orig -> origin/gh/coconutruben/85/orig 2025-12-04T09:23:39.8302633Z * [new branch] gh/coconutruben/86/base -> origin/gh/coconutruben/86/base 2025-12-04T09:23:39.8302815Z * [new branch] gh/coconutruben/86/head -> origin/gh/coconutruben/86/head 2025-12-04T09:23:39.8303089Z * [new branch] gh/coconutruben/86/orig -> origin/gh/coconutruben/86/orig 2025-12-04T09:23:39.8303320Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-12-04T09:23:39.8303490Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-12-04T09:23:39.8303660Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-12-04T09:23:39.8303841Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-12-04T09:23:39.8304010Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-12-04T09:23:39.8304183Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-12-04T09:23:39.8304353Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-12-04T09:23:39.8304519Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-12-04T09:23:39.8304671Z * [new branch] gh/d4l3k/1/base -> origin/gh/d4l3k/1/base 2025-12-04T09:23:39.8304822Z * [new branch] gh/d4l3k/1/head -> origin/gh/d4l3k/1/head 2025-12-04T09:23:39.8304977Z * [new branch] gh/d4l3k/2/base -> origin/gh/d4l3k/2/base 2025-12-04T09:23:39.8305146Z * [new branch] gh/d4l3k/2/head -> origin/gh/d4l3k/2/head 2025-12-04T09:23:39.8305450Z * [new branch] gh/d4l3k/2/orig -> origin/gh/d4l3k/2/orig 2025-12-04T09:23:39.8305592Z * [new branch] gh/d4l3k/3/base -> origin/gh/d4l3k/3/base 2025-12-04T09:23:39.8305721Z * [new branch] gh/d4l3k/3/head -> origin/gh/d4l3k/3/head 2025-12-04T09:23:39.8307252Z * [new branch] gh/d4l3k/3/orig -> origin/gh/d4l3k/3/orig 2025-12-04T09:23:39.8307485Z * [new branch] gh/d4l3k/4/base -> origin/gh/d4l3k/4/base 2025-12-04T09:23:39.8307664Z * [new branch] gh/d4l3k/4/head -> origin/gh/d4l3k/4/head 2025-12-04T09:23:39.8307819Z * [new branch] gh/d4l3k/4/orig -> origin/gh/d4l3k/4/orig 2025-12-04T09:23:39.8308013Z * [new branch] gh/d4l3k/5/base -> origin/gh/d4l3k/5/base 2025-12-04T09:23:39.8308164Z * [new branch] gh/d4l3k/5/orig -> origin/gh/d4l3k/5/orig 2025-12-04T09:23:39.8314711Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-12-04T09:23:39.8316861Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-12-04T09:23:39.8317222Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-12-04T09:23:39.8317421Z * [new branch] gh/davidberard98/399/base -> origin/gh/davidberard98/399/base 2025-12-04T09:23:39.8317585Z * [new branch] gh/davidberard98/399/head -> origin/gh/davidberard98/399/head 2025-12-04T09:23:39.8317750Z * [new branch] gh/davidberard98/399/orig -> origin/gh/davidberard98/399/orig 2025-12-04T09:23:39.8318038Z * [new branch] gh/desertfire/605/base -> origin/gh/desertfire/605/base 2025-12-04T09:23:39.8318407Z * [new branch] gh/desertfire/605/head -> origin/gh/desertfire/605/head 2025-12-04T09:23:39.8318585Z * [new branch] gh/desertfire/605/orig -> origin/gh/desertfire/605/orig 2025-12-04T09:23:39.8318737Z * [new branch] gh/desertfire/606/base -> origin/gh/desertfire/606/base 2025-12-04T09:23:39.8318899Z * [new branch] gh/desertfire/606/head -> origin/gh/desertfire/606/head 2025-12-04T09:23:39.8319057Z * [new branch] gh/desertfire/606/orig -> origin/gh/desertfire/606/orig 2025-12-04T09:23:39.8319213Z * [new branch] gh/desertfire/607/base -> origin/gh/desertfire/607/base 2025-12-04T09:23:39.8319362Z * [new branch] gh/desertfire/607/head -> origin/gh/desertfire/607/head 2025-12-04T09:23:39.8319513Z * [new branch] gh/desertfire/607/orig -> origin/gh/desertfire/607/orig 2025-12-04T09:23:39.8319670Z * [new branch] gh/desertfire/608/base -> origin/gh/desertfire/608/base 2025-12-04T09:23:39.8319820Z * [new branch] gh/desertfire/608/head -> origin/gh/desertfire/608/head 2025-12-04T09:23:39.8319979Z * [new branch] gh/desertfire/608/orig -> origin/gh/desertfire/608/orig 2025-12-04T09:23:39.8320259Z * [new branch] gh/desertfire/609/base -> origin/gh/desertfire/609/base 2025-12-04T09:23:39.8321012Z * [new branch] gh/desertfire/609/head -> origin/gh/desertfire/609/head 2025-12-04T09:23:39.8321247Z * [new branch] gh/desertfire/609/orig -> origin/gh/desertfire/609/orig 2025-12-04T09:23:39.8328935Z * [new branch] gh/desertfire/610/base -> origin/gh/desertfire/610/base 2025-12-04T09:23:39.8334051Z * [new branch] gh/desertfire/610/head -> origin/gh/desertfire/610/head 2025-12-04T09:23:39.8335341Z * [new branch] gh/desertfire/610/orig -> origin/gh/desertfire/610/orig 2025-12-04T09:23:39.8335545Z * [new branch] gh/desertfire/611/base -> origin/gh/desertfire/611/base 2025-12-04T09:23:39.8335857Z * [new branch] gh/desertfire/611/head -> origin/gh/desertfire/611/head 2025-12-04T09:23:39.8336105Z * [new branch] gh/desertfire/611/orig -> origin/gh/desertfire/611/orig 2025-12-04T09:23:39.8336315Z * [new branch] gh/desertfire/612/base -> origin/gh/desertfire/612/base 2025-12-04T09:23:39.8336707Z * [new branch] gh/desertfire/612/head -> origin/gh/desertfire/612/head 2025-12-04T09:23:39.8336868Z * [new branch] gh/desertfire/612/orig -> origin/gh/desertfire/612/orig 2025-12-04T09:23:39.8337018Z * [new branch] gh/desertfire/613/base -> origin/gh/desertfire/613/base 2025-12-04T09:23:39.8337164Z * [new branch] gh/desertfire/613/head -> origin/gh/desertfire/613/head 2025-12-04T09:23:39.8337317Z * [new branch] gh/desertfire/613/orig -> origin/gh/desertfire/613/orig 2025-12-04T09:23:39.8337467Z * [new branch] gh/desertfire/614/base -> origin/gh/desertfire/614/base 2025-12-04T09:23:39.8337627Z * [new branch] gh/desertfire/614/head -> origin/gh/desertfire/614/head 2025-12-04T09:23:39.8337777Z * [new branch] gh/desertfire/614/orig -> origin/gh/desertfire/614/orig 2025-12-04T09:23:39.8337928Z * [new branch] gh/desertfire/615/base -> origin/gh/desertfire/615/base 2025-12-04T09:23:39.8338090Z * [new branch] gh/desertfire/615/head -> origin/gh/desertfire/615/head 2025-12-04T09:23:39.8338242Z * [new branch] gh/desertfire/615/orig -> origin/gh/desertfire/615/orig 2025-12-04T09:23:39.8338410Z * [new branch] gh/desertfire/616/base -> origin/gh/desertfire/616/base 2025-12-04T09:23:39.8338563Z * [new branch] gh/desertfire/616/head -> origin/gh/desertfire/616/head 2025-12-04T09:23:39.8338711Z * [new branch] gh/desertfire/616/orig -> origin/gh/desertfire/616/orig 2025-12-04T09:23:39.8338871Z * [new branch] gh/desertfire/617/base -> origin/gh/desertfire/617/base 2025-12-04T09:23:39.8339343Z * [new branch] gh/desertfire/617/head -> origin/gh/desertfire/617/head 2025-12-04T09:23:39.8339890Z * [new branch] gh/desertfire/617/orig -> origin/gh/desertfire/617/orig 2025-12-04T09:23:39.8344037Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-12-04T09:23:39.8344366Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-12-04T09:23:39.8344675Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-12-04T09:23:39.8344852Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-12-04T09:23:39.8345093Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-12-04T09:23:39.8345261Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-12-04T09:23:39.8350305Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-12-04T09:23:39.8350639Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-12-04T09:23:39.8350817Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-12-04T09:23:39.8352670Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-12-04T09:23:39.8353001Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-12-04T09:23:39.8355219Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-12-04T09:23:39.8355570Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-12-04T09:23:39.8355744Z * [new branch] gh/drisspg/194/base -> origin/gh/drisspg/194/base 2025-12-04T09:23:39.8356167Z * [new branch] gh/drisspg/194/head -> origin/gh/drisspg/194/head 2025-12-04T09:23:39.8356968Z * [new branch] gh/drisspg/194/orig -> origin/gh/drisspg/194/orig 2025-12-04T09:23:39.8358215Z * [new branch] gh/drisspg/200/base -> origin/gh/drisspg/200/base 2025-12-04T09:23:39.8358915Z * [new branch] gh/drisspg/200/head -> origin/gh/drisspg/200/head 2025-12-04T09:23:39.8361356Z * [new branch] gh/drisspg/200/orig -> origin/gh/drisspg/200/orig 2025-12-04T09:23:39.8361737Z * [new branch] gh/drisspg/218/base -> origin/gh/drisspg/218/base 2025-12-04T09:23:39.8361984Z * [new branch] gh/drisspg/218/head -> origin/gh/drisspg/218/head 2025-12-04T09:23:39.8362155Z * [new branch] gh/drisspg/218/orig -> origin/gh/drisspg/218/orig 2025-12-04T09:23:39.8362493Z * [new branch] gh/drisspg/219/base -> origin/gh/drisspg/219/base 2025-12-04T09:23:39.8364922Z * [new branch] gh/drisspg/219/head -> origin/gh/drisspg/219/head 2025-12-04T09:23:39.8365285Z * [new branch] gh/drisspg/219/orig -> origin/gh/drisspg/219/orig 2025-12-04T09:23:39.8365466Z * [new branch] gh/drisspg/220/base -> origin/gh/drisspg/220/base 2025-12-04T09:23:39.8365610Z * [new branch] gh/drisspg/220/head -> origin/gh/drisspg/220/head 2025-12-04T09:23:39.8366160Z * [new branch] gh/drisspg/220/orig -> origin/gh/drisspg/220/orig 2025-12-04T09:23:39.8367609Z * [new branch] gh/drisspg/221/base -> origin/gh/drisspg/221/base 2025-12-04T09:23:39.8368137Z * [new branch] gh/drisspg/221/head -> origin/gh/drisspg/221/head 2025-12-04T09:23:39.8369278Z * [new branch] gh/drisspg/221/orig -> origin/gh/drisspg/221/orig 2025-12-04T09:23:39.8369843Z * [new branch] gh/drisspg/222/base -> origin/gh/drisspg/222/base 2025-12-04T09:23:39.8370033Z * [new branch] gh/drisspg/222/head -> origin/gh/drisspg/222/head 2025-12-04T09:23:39.8370343Z * [new branch] gh/drisspg/222/orig -> origin/gh/drisspg/222/orig 2025-12-04T09:23:39.8371794Z * [new branch] gh/drisspg/223/base -> origin/gh/drisspg/223/base 2025-12-04T09:23:39.8372511Z * [new branch] gh/drisspg/223/head -> origin/gh/drisspg/223/head 2025-12-04T09:23:39.8373013Z * [new branch] gh/drisspg/223/orig -> origin/gh/drisspg/223/orig 2025-12-04T09:23:39.8374354Z * [new branch] gh/drisspg/224/base -> origin/gh/drisspg/224/base 2025-12-04T09:23:39.8374524Z * [new branch] gh/drisspg/224/head -> origin/gh/drisspg/224/head 2025-12-04T09:23:39.8379861Z * [new branch] gh/drisspg/224/orig -> origin/gh/drisspg/224/orig 2025-12-04T09:23:39.8384388Z * [new branch] gh/drisspg/225/base -> origin/gh/drisspg/225/base 2025-12-04T09:23:39.8384563Z * [new branch] gh/drisspg/225/head -> origin/gh/drisspg/225/head 2025-12-04T09:23:39.8384721Z * [new branch] gh/drisspg/225/orig -> origin/gh/drisspg/225/orig 2025-12-04T09:23:39.8384861Z * [new branch] gh/drisspg/226/base -> origin/gh/drisspg/226/base 2025-12-04T09:23:39.8385006Z * [new branch] gh/drisspg/226/head -> origin/gh/drisspg/226/head 2025-12-04T09:23:39.8385153Z * [new branch] gh/drisspg/226/orig -> origin/gh/drisspg/226/orig 2025-12-04T09:23:39.8385287Z * [new branch] gh/drisspg/227/base -> origin/gh/drisspg/227/base 2025-12-04T09:23:39.8385419Z * [new branch] gh/drisspg/227/head -> origin/gh/drisspg/227/head 2025-12-04T09:23:39.8385559Z * [new branch] gh/drisspg/227/orig -> origin/gh/drisspg/227/orig 2025-12-04T09:23:39.8385689Z * [new branch] gh/drisspg/228/base -> origin/gh/drisspg/228/base 2025-12-04T09:23:39.8385827Z * [new branch] gh/drisspg/228/head -> origin/gh/drisspg/228/head 2025-12-04T09:23:39.8386139Z * [new branch] gh/drisspg/228/orig -> origin/gh/drisspg/228/orig 2025-12-04T09:23:39.8386281Z * [new branch] gh/drisspg/229/base -> origin/gh/drisspg/229/base 2025-12-04T09:23:39.8386423Z * [new branch] gh/drisspg/229/head -> origin/gh/drisspg/229/head 2025-12-04T09:23:39.8387524Z * [new branch] gh/drisspg/229/orig -> origin/gh/drisspg/229/orig 2025-12-04T09:23:39.8387806Z * [new branch] gh/drisspg/230/base -> origin/gh/drisspg/230/base 2025-12-04T09:23:39.8388885Z * [new branch] gh/drisspg/230/head -> origin/gh/drisspg/230/head 2025-12-04T09:23:39.8389043Z * [new branch] gh/drisspg/230/orig -> origin/gh/drisspg/230/orig 2025-12-04T09:23:39.8391693Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-12-04T09:23:39.8392081Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-12-04T09:23:39.8392384Z * [new branch] gh/dzmitry-huba/1/base -> origin/gh/dzmitry-huba/1/base 2025-12-04T09:23:39.8393373Z * [new branch] gh/dzmitry-huba/1/head -> origin/gh/dzmitry-huba/1/head 2025-12-04T09:23:39.8394535Z * [new branch] gh/dzmitry-huba/12/base -> origin/gh/dzmitry-huba/12/base 2025-12-04T09:23:39.8395241Z * [new branch] gh/dzmitry-huba/12/head -> origin/gh/dzmitry-huba/12/head 2025-12-04T09:23:39.8395626Z * [new branch] gh/dzmitry-huba/12/orig -> origin/gh/dzmitry-huba/12/orig 2025-12-04T09:23:39.8396960Z * [new branch] gh/dzmitry-huba/13/base -> origin/gh/dzmitry-huba/13/base 2025-12-04T09:23:39.8397449Z * [new branch] gh/dzmitry-huba/13/head -> origin/gh/dzmitry-huba/13/head 2025-12-04T09:23:39.8398339Z * [new branch] gh/dzmitry-huba/13/orig -> origin/gh/dzmitry-huba/13/orig 2025-12-04T09:23:39.8399252Z * [new branch] gh/dzmitry-huba/14/base -> origin/gh/dzmitry-huba/14/base 2025-12-04T09:23:39.8399772Z * [new branch] gh/dzmitry-huba/14/head -> origin/gh/dzmitry-huba/14/head 2025-12-04T09:23:39.8401146Z * [new branch] gh/dzmitry-huba/14/orig -> origin/gh/dzmitry-huba/14/orig 2025-12-04T09:23:39.8401659Z * [new branch] gh/dzmitry-huba/15/base -> origin/gh/dzmitry-huba/15/base 2025-12-04T09:23:39.8402580Z * [new branch] gh/dzmitry-huba/15/head -> origin/gh/dzmitry-huba/15/head 2025-12-04T09:23:39.8402993Z * [new branch] gh/dzmitry-huba/15/orig -> origin/gh/dzmitry-huba/15/orig 2025-12-04T09:23:39.8404329Z * [new branch] gh/dzmitry-huba/16/base -> origin/gh/dzmitry-huba/16/base 2025-12-04T09:23:39.8405782Z * [new branch] gh/dzmitry-huba/16/head -> origin/gh/dzmitry-huba/16/head 2025-12-04T09:23:39.8406246Z * [new branch] gh/dzmitry-huba/16/orig -> origin/gh/dzmitry-huba/16/orig 2025-12-04T09:23:39.8406445Z * [new branch] gh/dzmitry-huba/17/base -> origin/gh/dzmitry-huba/17/base 2025-12-04T09:23:39.8407220Z * [new branch] gh/dzmitry-huba/17/head -> origin/gh/dzmitry-huba/17/head 2025-12-04T09:23:39.8407829Z * [new branch] gh/dzmitry-huba/17/orig -> origin/gh/dzmitry-huba/17/orig 2025-12-04T09:23:39.8408898Z * [new branch] gh/dzmitry-huba/2/base -> origin/gh/dzmitry-huba/2/base 2025-12-04T09:23:39.8409219Z * [new branch] gh/dzmitry-huba/2/head -> origin/gh/dzmitry-huba/2/head 2025-12-04T09:23:39.8413753Z * [new branch] gh/dzmitry-huba/3/base -> origin/gh/dzmitry-huba/3/base 2025-12-04T09:23:39.8413910Z * [new branch] gh/dzmitry-huba/3/head -> origin/gh/dzmitry-huba/3/head 2025-12-04T09:23:39.8414082Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-12-04T09:23:39.8414429Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-12-04T09:23:39.8414588Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-12-04T09:23:39.8416597Z * [new branch] gh/eellison/822/base -> origin/gh/eellison/822/base 2025-12-04T09:23:39.8416911Z * [new branch] gh/eellison/822/head -> origin/gh/eellison/822/head 2025-12-04T09:23:39.8417064Z * [new branch] gh/eellison/822/orig -> origin/gh/eellison/822/orig 2025-12-04T09:23:39.8417225Z * [new branch] gh/eellison/823/base -> origin/gh/eellison/823/base 2025-12-04T09:23:39.8417678Z * [new branch] gh/eellison/823/head -> origin/gh/eellison/823/head 2025-12-04T09:23:39.8418416Z * [new branch] gh/eellison/823/orig -> origin/gh/eellison/823/orig 2025-12-04T09:23:39.8419897Z * [new branch] gh/eellison/862/base -> origin/gh/eellison/862/base 2025-12-04T09:23:39.8420156Z * [new branch] gh/eellison/862/head -> origin/gh/eellison/862/head 2025-12-04T09:23:39.8420673Z * [new branch] gh/eellison/862/orig -> origin/gh/eellison/862/orig 2025-12-04T09:23:39.8421843Z * [new branch] gh/eellison/863/base -> origin/gh/eellison/863/base 2025-12-04T09:23:39.8422356Z * [new branch] gh/eellison/863/head -> origin/gh/eellison/863/head 2025-12-04T09:23:39.8423315Z * [new branch] gh/eellison/863/orig -> origin/gh/eellison/863/orig 2025-12-04T09:23:39.8423886Z * [new branch] gh/eellison/864/base -> origin/gh/eellison/864/base 2025-12-04T09:23:39.8424852Z * [new branch] gh/eellison/864/head -> origin/gh/eellison/864/head 2025-12-04T09:23:39.8425069Z * [new branch] gh/eellison/864/orig -> origin/gh/eellison/864/orig 2025-12-04T09:23:39.8426505Z * [new branch] gh/eellison/865/base -> origin/gh/eellison/865/base 2025-12-04T09:23:39.8426795Z * [new branch] gh/eellison/865/head -> origin/gh/eellison/865/head 2025-12-04T09:23:39.8428061Z * [new branch] gh/eellison/865/orig -> origin/gh/eellison/865/orig 2025-12-04T09:23:39.8428950Z * [new branch] gh/eellison/866/base -> origin/gh/eellison/866/base 2025-12-04T09:23:39.8429295Z * [new branch] gh/eellison/866/head -> origin/gh/eellison/866/head 2025-12-04T09:23:39.8430261Z * [new branch] gh/eellison/866/orig -> origin/gh/eellison/866/orig 2025-12-04T09:23:39.8431343Z * [new branch] gh/eellison/867/base -> origin/gh/eellison/867/base 2025-12-04T09:23:39.8431820Z * [new branch] gh/eellison/867/head -> origin/gh/eellison/867/head 2025-12-04T09:23:39.8432782Z * [new branch] gh/eellison/867/orig -> origin/gh/eellison/867/orig 2025-12-04T09:23:39.8433890Z * [new branch] gh/eellison/868/base -> origin/gh/eellison/868/base 2025-12-04T09:23:39.8434842Z * [new branch] gh/eellison/868/head -> origin/gh/eellison/868/head 2025-12-04T09:23:39.8435273Z * [new branch] gh/eellison/868/orig -> origin/gh/eellison/868/orig 2025-12-04T09:23:39.8436581Z * [new branch] gh/eellison/869/base -> origin/gh/eellison/869/base 2025-12-04T09:23:39.8436871Z * [new branch] gh/eellison/869/head -> origin/gh/eellison/869/head 2025-12-04T09:23:39.8437786Z * [new branch] gh/eellison/869/orig -> origin/gh/eellison/869/orig 2025-12-04T09:23:39.8438849Z * [new branch] gh/eellison/870/base -> origin/gh/eellison/870/base 2025-12-04T09:23:39.8439331Z * [new branch] gh/eellison/870/head -> origin/gh/eellison/870/head 2025-12-04T09:23:39.8441310Z * [new branch] gh/eellison/870/orig -> origin/gh/eellison/870/orig 2025-12-04T09:23:39.8442122Z * [new branch] gh/eellison/871/base -> origin/gh/eellison/871/base 2025-12-04T09:23:39.8442688Z * [new branch] gh/eellison/871/head -> origin/gh/eellison/871/head 2025-12-04T09:23:39.8442841Z * [new branch] gh/eellison/871/orig -> origin/gh/eellison/871/orig 2025-12-04T09:23:39.8449957Z * [new branch] gh/eellison/872/base -> origin/gh/eellison/872/base 2025-12-04T09:23:39.8450156Z * [new branch] gh/eellison/872/head -> origin/gh/eellison/872/head 2025-12-04T09:23:39.8450330Z * [new branch] gh/eellison/872/orig -> origin/gh/eellison/872/orig 2025-12-04T09:23:39.8450484Z * [new branch] gh/eellison/873/base -> origin/gh/eellison/873/base 2025-12-04T09:23:39.8450640Z * [new branch] gh/eellison/873/head -> origin/gh/eellison/873/head 2025-12-04T09:23:39.8450799Z * [new branch] gh/eellison/873/orig -> origin/gh/eellison/873/orig 2025-12-04T09:23:39.8450952Z * [new branch] gh/eellison/874/base -> origin/gh/eellison/874/base 2025-12-04T09:23:39.8451146Z * [new branch] gh/eellison/874/head -> origin/gh/eellison/874/head 2025-12-04T09:23:39.8451310Z * [new branch] gh/eellison/874/orig -> origin/gh/eellison/874/orig 2025-12-04T09:23:39.8451471Z * [new branch] gh/eellison/875/base -> origin/gh/eellison/875/base 2025-12-04T09:23:39.8451641Z * [new branch] gh/eellison/875/head -> origin/gh/eellison/875/head 2025-12-04T09:23:39.8452933Z * [new branch] gh/eellison/875/orig -> origin/gh/eellison/875/orig 2025-12-04T09:23:39.8458882Z * [new branch] gh/eellison/876/base -> origin/gh/eellison/876/base 2025-12-04T09:23:39.8459212Z * [new branch] gh/eellison/876/head -> origin/gh/eellison/876/head 2025-12-04T09:23:39.8459376Z * [new branch] gh/eellison/876/orig -> origin/gh/eellison/876/orig 2025-12-04T09:23:39.8459548Z * [new branch] gh/eellison/877/base -> origin/gh/eellison/877/base 2025-12-04T09:23:39.8459766Z * [new branch] gh/eellison/877/head -> origin/gh/eellison/877/head 2025-12-04T09:23:39.8460011Z * [new branch] gh/eellison/877/orig -> origin/gh/eellison/877/orig 2025-12-04T09:23:39.8460159Z * [new branch] gh/eellison/878/base -> origin/gh/eellison/878/base 2025-12-04T09:23:39.8460412Z * [new branch] gh/eellison/878/head -> origin/gh/eellison/878/head 2025-12-04T09:23:39.8460618Z * [new branch] gh/eellison/878/orig -> origin/gh/eellison/878/orig 2025-12-04T09:23:39.8460752Z * [new branch] gh/eellison/879/base -> origin/gh/eellison/879/base 2025-12-04T09:23:39.8460890Z * [new branch] gh/eellison/879/head -> origin/gh/eellison/879/head 2025-12-04T09:23:39.8461036Z * [new branch] gh/eellison/879/orig -> origin/gh/eellison/879/orig 2025-12-04T09:23:39.8466814Z * [new branch] gh/eellison/880/base -> origin/gh/eellison/880/base 2025-12-04T09:23:39.8467148Z * [new branch] gh/eellison/880/head -> origin/gh/eellison/880/head 2025-12-04T09:23:39.8467368Z * [new branch] gh/eellison/880/orig -> origin/gh/eellison/880/orig 2025-12-04T09:23:39.8467544Z * [new branch] gh/eellison/881/base -> origin/gh/eellison/881/base 2025-12-04T09:23:39.8467714Z * [new branch] gh/eellison/881/head -> origin/gh/eellison/881/head 2025-12-04T09:23:39.8467851Z * [new branch] gh/eellison/881/orig -> origin/gh/eellison/881/orig 2025-12-04T09:23:39.8468127Z * [new branch] gh/eellison/882/base -> origin/gh/eellison/882/base 2025-12-04T09:23:39.8468796Z * [new branch] gh/eellison/882/head -> origin/gh/eellison/882/head 2025-12-04T09:23:39.8469348Z * [new branch] gh/eellison/882/orig -> origin/gh/eellison/882/orig 2025-12-04T09:23:39.8469502Z * [new branch] gh/eellison/883/base -> origin/gh/eellison/883/base 2025-12-04T09:23:39.8469735Z * [new branch] gh/eellison/883/head -> origin/gh/eellison/883/head 2025-12-04T09:23:39.8469946Z * [new branch] gh/eellison/883/orig -> origin/gh/eellison/883/orig 2025-12-04T09:23:39.8471290Z * [new branch] gh/eellison/884/base -> origin/gh/eellison/884/base 2025-12-04T09:23:39.8471562Z * [new branch] gh/eellison/884/head -> origin/gh/eellison/884/head 2025-12-04T09:23:39.8471937Z * [new branch] gh/eellison/884/orig -> origin/gh/eellison/884/orig 2025-12-04T09:23:39.8474115Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-12-04T09:23:39.8474315Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-12-04T09:23:39.8479362Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-12-04T09:23:39.8479536Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-12-04T09:23:39.8479672Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-12-04T09:23:39.8479829Z * [new branch] gh/etaf/156/base -> origin/gh/etaf/156/base 2025-12-04T09:23:39.8479963Z * [new branch] gh/etaf/156/head -> origin/gh/etaf/156/head 2025-12-04T09:23:39.8480104Z * [new branch] gh/etaf/156/orig -> origin/gh/etaf/156/orig 2025-12-04T09:23:39.8480243Z * [new branch] gh/etaf/157/base -> origin/gh/etaf/157/base 2025-12-04T09:23:39.8480568Z * [new branch] gh/etaf/157/head -> origin/gh/etaf/157/head 2025-12-04T09:23:39.8486080Z * [new branch] gh/etaf/157/orig -> origin/gh/etaf/157/orig 2025-12-04T09:23:39.8488784Z * [new branch] gh/etaf/158/base -> origin/gh/etaf/158/base 2025-12-04T09:23:39.8489276Z * [new branch] gh/etaf/158/head -> origin/gh/etaf/158/head 2025-12-04T09:23:39.8489449Z * [new branch] gh/etaf/158/orig -> origin/gh/etaf/158/orig 2025-12-04T09:23:39.8489608Z * [new branch] gh/etaf/159/base -> origin/gh/etaf/159/base 2025-12-04T09:23:39.8489765Z * [new branch] gh/etaf/159/head -> origin/gh/etaf/159/head 2025-12-04T09:23:39.8489912Z * [new branch] gh/etaf/159/orig -> origin/gh/etaf/159/orig 2025-12-04T09:23:39.8490040Z * [new branch] gh/etaf/160/base -> origin/gh/etaf/160/base 2025-12-04T09:23:39.8490166Z * [new branch] gh/etaf/160/head -> origin/gh/etaf/160/head 2025-12-04T09:23:39.8490298Z * [new branch] gh/etaf/160/orig -> origin/gh/etaf/160/orig 2025-12-04T09:23:39.8490438Z * [new branch] gh/etaf/161/base -> origin/gh/etaf/161/base 2025-12-04T09:23:39.8492816Z * [new branch] gh/etaf/161/head -> origin/gh/etaf/161/head 2025-12-04T09:23:39.8492997Z * [new branch] gh/etaf/161/orig -> origin/gh/etaf/161/orig 2025-12-04T09:23:39.8493146Z * [new branch] gh/etaf/166/base -> origin/gh/etaf/166/base 2025-12-04T09:23:39.8493276Z * [new branch] gh/etaf/166/head -> origin/gh/etaf/166/head 2025-12-04T09:23:39.8493482Z * [new branch] gh/etaf/166/orig -> origin/gh/etaf/166/orig 2025-12-04T09:23:39.8499592Z * [new branch] gh/etaf/167/base -> origin/gh/etaf/167/base 2025-12-04T09:23:39.8505135Z * [new branch] gh/etaf/167/head -> origin/gh/etaf/167/head 2025-12-04T09:23:39.8505308Z * [new branch] gh/etaf/167/orig -> origin/gh/etaf/167/orig 2025-12-04T09:23:39.8505598Z * [new branch] gh/etaf/168/base -> origin/gh/etaf/168/base 2025-12-04T09:23:39.8505726Z * [new branch] gh/etaf/168/head -> origin/gh/etaf/168/head 2025-12-04T09:23:39.8505853Z * [new branch] gh/etaf/168/orig -> origin/gh/etaf/168/orig 2025-12-04T09:23:39.8506067Z * [new branch] gh/etaf/172/base -> origin/gh/etaf/172/base 2025-12-04T09:23:39.8506191Z * [new branch] gh/etaf/172/head -> origin/gh/etaf/172/head 2025-12-04T09:23:39.8506312Z * [new branch] gh/etaf/172/orig -> origin/gh/etaf/172/orig 2025-12-04T09:23:39.8506436Z * [new branch] gh/etaf/173/base -> origin/gh/etaf/173/base 2025-12-04T09:23:39.8506557Z * [new branch] gh/etaf/173/head -> origin/gh/etaf/173/head 2025-12-04T09:23:39.8506683Z * [new branch] gh/etaf/173/orig -> origin/gh/etaf/173/orig 2025-12-04T09:23:39.8506808Z * [new branch] gh/etaf/174/base -> origin/gh/etaf/174/base 2025-12-04T09:23:39.8506930Z * [new branch] gh/etaf/174/head -> origin/gh/etaf/174/head 2025-12-04T09:23:39.8507060Z * [new branch] gh/etaf/175/base -> origin/gh/etaf/175/base 2025-12-04T09:23:39.8507346Z * [new branch] gh/etaf/175/head -> origin/gh/etaf/175/head 2025-12-04T09:23:39.8507506Z * [new branch] gh/etaf/175/orig -> origin/gh/etaf/175/orig 2025-12-04T09:23:39.8507627Z * [new branch] gh/etaf/176/base -> origin/gh/etaf/176/base 2025-12-04T09:23:39.8508358Z * [new branch] gh/etaf/176/head -> origin/gh/etaf/176/head 2025-12-04T09:23:39.8512424Z * [new branch] gh/etaf/176/orig -> origin/gh/etaf/176/orig 2025-12-04T09:23:39.8512766Z * [new branch] gh/etaf/177/base -> origin/gh/etaf/177/base 2025-12-04T09:23:39.8512989Z * [new branch] gh/etaf/177/head -> origin/gh/etaf/177/head 2025-12-04T09:23:39.8513176Z * [new branch] gh/etaf/177/orig -> origin/gh/etaf/177/orig 2025-12-04T09:23:39.8513386Z * [new branch] gh/etaf/178/base -> origin/gh/etaf/178/base 2025-12-04T09:23:39.8513562Z * [new branch] gh/etaf/178/head -> origin/gh/etaf/178/head 2025-12-04T09:23:39.8514292Z * [new branch] gh/etaf/178/orig -> origin/gh/etaf/178/orig 2025-12-04T09:23:39.8515447Z * [new branch] gh/etaf/179/base -> origin/gh/etaf/179/base 2025-12-04T09:23:39.8515939Z * [new branch] gh/etaf/179/head -> origin/gh/etaf/179/head 2025-12-04T09:23:39.8516651Z * [new branch] gh/etaf/179/orig -> origin/gh/etaf/179/orig 2025-12-04T09:23:39.8517653Z * [new branch] gh/etaf/180/base -> origin/gh/etaf/180/base 2025-12-04T09:23:39.8520151Z * [new branch] gh/etaf/180/head -> origin/gh/etaf/180/head 2025-12-04T09:23:39.8520491Z * [new branch] gh/etaf/180/orig -> origin/gh/etaf/180/orig 2025-12-04T09:23:39.8520684Z * [new branch] gh/exclamaforte/1/base -> origin/gh/exclamaforte/1/base 2025-12-04T09:23:39.8521749Z * [new branch] gh/exclamaforte/1/head -> origin/gh/exclamaforte/1/head 2025-12-04T09:23:39.8522054Z * [new branch] gh/exclamaforte/2/base -> origin/gh/exclamaforte/2/base 2025-12-04T09:23:39.8524773Z * [new branch] gh/exclamaforte/2/head -> origin/gh/exclamaforte/2/head 2025-12-04T09:23:39.8524960Z * [new branch] gh/exclamaforte/3/base -> origin/gh/exclamaforte/3/base 2025-12-04T09:23:39.8525120Z * [new branch] gh/exclamaforte/3/head -> origin/gh/exclamaforte/3/head 2025-12-04T09:23:39.8525586Z * [new branch] gh/exclamaforte/4/base -> origin/gh/exclamaforte/4/base 2025-12-04T09:23:39.8526335Z * [new branch] gh/exclamaforte/4/head -> origin/gh/exclamaforte/4/head 2025-12-04T09:23:39.8530757Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-12-04T09:23:39.8530925Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-12-04T09:23:39.8531212Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-12-04T09:23:39.8531362Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-12-04T09:23:39.8531494Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-12-04T09:23:39.8531634Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-12-04T09:23:39.8532032Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-12-04T09:23:39.8533257Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-12-04T09:23:39.8533546Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-12-04T09:23:39.8536209Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-12-04T09:23:39.8536614Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-12-04T09:23:39.8536784Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-12-04T09:23:39.8536928Z * [new branch] gh/ezyang/3139/base -> origin/gh/ezyang/3139/base 2025-12-04T09:23:39.8537333Z * [new branch] gh/ezyang/3139/head -> origin/gh/ezyang/3139/head 2025-12-04T09:23:39.8538097Z * [new branch] gh/ezyang/3139/orig -> origin/gh/ezyang/3139/orig 2025-12-04T09:23:39.8544135Z * [new branch] gh/ezyang/3140/base -> origin/gh/ezyang/3140/base 2025-12-04T09:23:39.8548468Z * [new branch] gh/ezyang/3140/head -> origin/gh/ezyang/3140/head 2025-12-04T09:23:39.8548952Z * [new branch] gh/ezyang/3140/orig -> origin/gh/ezyang/3140/orig 2025-12-04T09:23:39.8549237Z * [new branch] gh/ezyang/3143/base -> origin/gh/ezyang/3143/base 2025-12-04T09:23:39.8549400Z * [new branch] gh/ezyang/3143/head -> origin/gh/ezyang/3143/head 2025-12-04T09:23:39.8549559Z * [new branch] gh/ezyang/3143/orig -> origin/gh/ezyang/3143/orig 2025-12-04T09:23:39.8549813Z * [new branch] gh/ezyang/3144/base -> origin/gh/ezyang/3144/base 2025-12-04T09:23:39.8549964Z * [new branch] gh/ezyang/3144/head -> origin/gh/ezyang/3144/head 2025-12-04T09:23:39.8550604Z * [new branch] gh/ezyang/3144/orig -> origin/gh/ezyang/3144/orig 2025-12-04T09:23:39.8550800Z * [new branch] gh/ezyang/3167/base -> origin/gh/ezyang/3167/base 2025-12-04T09:23:39.8550967Z * [new branch] gh/ezyang/3167/head -> origin/gh/ezyang/3167/head 2025-12-04T09:23:39.8551117Z * [new branch] gh/ezyang/3167/orig -> origin/gh/ezyang/3167/orig 2025-12-04T09:23:39.8551258Z * [new branch] gh/ezyang/3173/base -> origin/gh/ezyang/3173/base 2025-12-04T09:23:39.8551399Z * [new branch] gh/ezyang/3173/head -> origin/gh/ezyang/3173/head 2025-12-04T09:23:39.8551557Z * [new branch] gh/ezyang/3173/orig -> origin/gh/ezyang/3173/orig 2025-12-04T09:23:39.8551695Z * [new branch] gh/ezyang/3175/base -> origin/gh/ezyang/3175/base 2025-12-04T09:23:39.8551846Z * [new branch] gh/ezyang/3175/head -> origin/gh/ezyang/3175/head 2025-12-04T09:23:39.8551984Z * [new branch] gh/ezyang/3175/orig -> origin/gh/ezyang/3175/orig 2025-12-04T09:23:39.8553001Z * [new branch] gh/ezyang/3182/base -> origin/gh/ezyang/3182/base 2025-12-04T09:23:39.8553410Z * [new branch] gh/ezyang/3182/head -> origin/gh/ezyang/3182/head 2025-12-04T09:23:39.8556757Z * [new branch] gh/ezyang/3182/orig -> origin/gh/ezyang/3182/orig 2025-12-04T09:23:39.8556957Z * [new branch] gh/ezyang/3185/base -> origin/gh/ezyang/3185/base 2025-12-04T09:23:39.8557326Z * [new branch] gh/ezyang/3185/head -> origin/gh/ezyang/3185/head 2025-12-04T09:23:39.8557481Z * [new branch] gh/ezyang/3185/orig -> origin/gh/ezyang/3185/orig 2025-12-04T09:23:39.8557806Z * [new branch] gh/ezyang/3189/base -> origin/gh/ezyang/3189/base 2025-12-04T09:23:39.8557972Z * [new branch] gh/ezyang/3189/head -> origin/gh/ezyang/3189/head 2025-12-04T09:23:39.8559132Z * [new branch] gh/ezyang/3189/orig -> origin/gh/ezyang/3189/orig 2025-12-04T09:23:39.8560200Z * [new branch] gh/ezyang/3191/base -> origin/gh/ezyang/3191/base 2025-12-04T09:23:39.8560513Z * [new branch] gh/ezyang/3191/head -> origin/gh/ezyang/3191/head 2025-12-04T09:23:39.8561319Z * [new branch] gh/ezyang/3191/orig -> origin/gh/ezyang/3191/orig 2025-12-04T09:23:39.8565495Z * [new branch] gh/ezyang/3192/base -> origin/gh/ezyang/3192/base 2025-12-04T09:23:39.8565854Z * [new branch] gh/ezyang/3192/head -> origin/gh/ezyang/3192/head 2025-12-04T09:23:39.8566098Z * [new branch] gh/ezyang/3192/orig -> origin/gh/ezyang/3192/orig 2025-12-04T09:23:39.8566254Z * [new branch] gh/ezyang/3193/base -> origin/gh/ezyang/3193/base 2025-12-04T09:23:39.8566409Z * [new branch] gh/ezyang/3193/head -> origin/gh/ezyang/3193/head 2025-12-04T09:23:39.8566697Z * [new branch] gh/ezyang/3193/orig -> origin/gh/ezyang/3193/orig 2025-12-04T09:23:39.8567427Z * [new branch] gh/ezyang/3194/base -> origin/gh/ezyang/3194/base 2025-12-04T09:23:39.8568084Z * [new branch] gh/ezyang/3194/head -> origin/gh/ezyang/3194/head 2025-12-04T09:23:39.8568537Z * [new branch] gh/ezyang/3194/orig -> origin/gh/ezyang/3194/orig 2025-12-04T09:23:39.8571026Z * [new branch] gh/ezyang/3195/base -> origin/gh/ezyang/3195/base 2025-12-04T09:23:39.8571378Z * [new branch] gh/ezyang/3195/head -> origin/gh/ezyang/3195/head 2025-12-04T09:23:39.8571600Z * [new branch] gh/ezyang/3195/orig -> origin/gh/ezyang/3195/orig 2025-12-04T09:23:39.8571929Z * [new branch] gh/ezyang/3196/base -> origin/gh/ezyang/3196/base 2025-12-04T09:23:39.8572955Z * [new branch] gh/ezyang/3196/head -> origin/gh/ezyang/3196/head 2025-12-04T09:23:39.8573256Z * [new branch] gh/ezyang/3196/orig -> origin/gh/ezyang/3196/orig 2025-12-04T09:23:39.8576679Z * [new branch] gh/ezyang/3197/base -> origin/gh/ezyang/3197/base 2025-12-04T09:23:39.8582109Z * [new branch] gh/ezyang/3197/head -> origin/gh/ezyang/3197/head 2025-12-04T09:23:39.8584348Z * [new branch] gh/ezyang/3197/orig -> origin/gh/ezyang/3197/orig 2025-12-04T09:23:39.8584613Z * [new branch] gh/ezyang/3198/base -> origin/gh/ezyang/3198/base 2025-12-04T09:23:39.8590183Z * [new branch] gh/ezyang/3198/head -> origin/gh/ezyang/3198/head 2025-12-04T09:23:39.8592020Z * [new branch] gh/ezyang/3198/orig -> origin/gh/ezyang/3198/orig 2025-12-04T09:23:39.8592188Z * [new branch] gh/ezyang/3199/base -> origin/gh/ezyang/3199/base 2025-12-04T09:23:39.8592319Z * [new branch] gh/ezyang/3199/head -> origin/gh/ezyang/3199/head 2025-12-04T09:23:39.8592449Z * [new branch] gh/ezyang/3199/orig -> origin/gh/ezyang/3199/orig 2025-12-04T09:23:39.8592583Z * [new branch] gh/ezyang/3200/base -> origin/gh/ezyang/3200/base 2025-12-04T09:23:39.8592863Z * [new branch] gh/ezyang/3200/head -> origin/gh/ezyang/3200/head 2025-12-04T09:23:39.8593004Z * [new branch] gh/ezyang/3200/orig -> origin/gh/ezyang/3200/orig 2025-12-04T09:23:39.8593133Z * [new branch] gh/ezyang/3201/base -> origin/gh/ezyang/3201/base 2025-12-04T09:23:39.8593334Z * [new branch] gh/ezyang/3201/head -> origin/gh/ezyang/3201/head 2025-12-04T09:23:39.8593474Z * [new branch] gh/ezyang/3201/orig -> origin/gh/ezyang/3201/orig 2025-12-04T09:23:39.8593609Z * [new branch] gh/ezyang/3202/base -> origin/gh/ezyang/3202/base 2025-12-04T09:23:39.8593741Z * [new branch] gh/ezyang/3202/head -> origin/gh/ezyang/3202/head 2025-12-04T09:23:39.8593887Z * [new branch] gh/ezyang/3202/orig -> origin/gh/ezyang/3202/orig 2025-12-04T09:23:39.8594015Z * [new branch] gh/ezyang/3203/base -> origin/gh/ezyang/3203/base 2025-12-04T09:23:39.8594148Z * [new branch] gh/ezyang/3203/head -> origin/gh/ezyang/3203/head 2025-12-04T09:23:39.8594274Z * [new branch] gh/ezyang/3203/orig -> origin/gh/ezyang/3203/orig 2025-12-04T09:23:39.8594406Z * [new branch] gh/ezyang/3204/base -> origin/gh/ezyang/3204/base 2025-12-04T09:23:39.8594545Z * [new branch] gh/ezyang/3204/head -> origin/gh/ezyang/3204/head 2025-12-04T09:23:39.8594674Z * [new branch] gh/ezyang/3204/orig -> origin/gh/ezyang/3204/orig 2025-12-04T09:23:39.8594815Z * [new branch] gh/ezyang/3205/base -> origin/gh/ezyang/3205/base 2025-12-04T09:23:39.8596947Z * [new branch] gh/ezyang/3205/head -> origin/gh/ezyang/3205/head 2025-12-04T09:23:39.8597488Z * [new branch] gh/ezyang/3205/orig -> origin/gh/ezyang/3205/orig 2025-12-04T09:23:39.8597692Z * [new branch] gh/ezyang/3206/base -> origin/gh/ezyang/3206/base 2025-12-04T09:23:39.8597855Z * [new branch] gh/ezyang/3206/head -> origin/gh/ezyang/3206/head 2025-12-04T09:23:39.8598004Z * [new branch] gh/ezyang/3206/orig -> origin/gh/ezyang/3206/orig 2025-12-04T09:23:39.8598184Z * [new branch] gh/ezyang/3207/base -> origin/gh/ezyang/3207/base 2025-12-04T09:23:39.8598342Z * [new branch] gh/ezyang/3207/head -> origin/gh/ezyang/3207/head 2025-12-04T09:23:39.8598515Z * [new branch] gh/ezyang/3207/orig -> origin/gh/ezyang/3207/orig 2025-12-04T09:23:39.8600729Z * [new branch] gh/ezyang/3208/base -> origin/gh/ezyang/3208/base 2025-12-04T09:23:39.8600898Z * [new branch] gh/ezyang/3208/head -> origin/gh/ezyang/3208/head 2025-12-04T09:23:39.8601050Z * [new branch] gh/ezyang/3208/orig -> origin/gh/ezyang/3208/orig 2025-12-04T09:23:39.8608388Z * [new branch] gh/ezyang/3209/base -> origin/gh/ezyang/3209/base 2025-12-04T09:23:39.8613694Z * [new branch] gh/ezyang/3209/head -> origin/gh/ezyang/3209/head 2025-12-04T09:23:39.8615510Z * [new branch] gh/ezyang/3209/orig -> origin/gh/ezyang/3209/orig 2025-12-04T09:23:39.8615706Z * [new branch] gh/fadara01/3/base -> origin/gh/fadara01/3/base 2025-12-04T09:23:39.8615861Z * [new branch] gh/fadara01/3/head -> origin/gh/fadara01/3/head 2025-12-04T09:23:39.8615994Z * [new branch] gh/fadara01/3/orig -> origin/gh/fadara01/3/orig 2025-12-04T09:23:39.8616142Z * [new branch] gh/fadara01/5/base -> origin/gh/fadara01/5/base 2025-12-04T09:23:39.8616283Z * [new branch] gh/fadara01/5/head -> origin/gh/fadara01/5/head 2025-12-04T09:23:39.8616430Z * [new branch] gh/fadara01/5/orig -> origin/gh/fadara01/5/orig 2025-12-04T09:23:39.8616733Z * [new branch] gh/fadara01/6/base -> origin/gh/fadara01/6/base 2025-12-04T09:23:39.8616874Z * [new branch] gh/fadara01/6/head -> origin/gh/fadara01/6/head 2025-12-04T09:23:39.8617023Z * [new branch] gh/fadara01/6/orig -> origin/gh/fadara01/6/orig 2025-12-04T09:23:39.8617238Z * [new branch] gh/fadara01/7/base -> origin/gh/fadara01/7/base 2025-12-04T09:23:39.8617387Z * [new branch] gh/fadara01/7/head -> origin/gh/fadara01/7/head 2025-12-04T09:23:39.8617523Z * [new branch] gh/fadara01/7/orig -> origin/gh/fadara01/7/orig 2025-12-04T09:23:39.8617662Z * [new branch] gh/fadara01/8/base -> origin/gh/fadara01/8/base 2025-12-04T09:23:39.8617805Z * [new branch] gh/fadara01/8/head -> origin/gh/fadara01/8/head 2025-12-04T09:23:39.8619947Z * [new branch] gh/fadara01/8/orig -> origin/gh/fadara01/8/orig 2025-12-04T09:23:39.8620176Z * [new branch] gh/fadara01/9/base -> origin/gh/fadara01/9/base 2025-12-04T09:23:39.8620351Z * [new branch] gh/fadara01/9/head -> origin/gh/fadara01/9/head 2025-12-04T09:23:39.8620500Z * [new branch] gh/fadara01/9/orig -> origin/gh/fadara01/9/orig 2025-12-04T09:23:39.8620651Z * [new branch] gh/fduwjj/182/base -> origin/gh/fduwjj/182/base 2025-12-04T09:23:39.8620854Z * [new branch] gh/fduwjj/182/head -> origin/gh/fduwjj/182/head 2025-12-04T09:23:39.8624510Z * [new branch] gh/fduwjj/182/orig -> origin/gh/fduwjj/182/orig 2025-12-04T09:23:39.8624677Z * [new branch] gh/fduwjj/211/base -> origin/gh/fduwjj/211/base 2025-12-04T09:23:39.8624828Z * [new branch] gh/fduwjj/211/head -> origin/gh/fduwjj/211/head 2025-12-04T09:23:39.8625031Z * [new branch] gh/fduwjj/211/orig -> origin/gh/fduwjj/211/orig 2025-12-04T09:23:39.8625189Z * [new branch] gh/fduwjj/212/base -> origin/gh/fduwjj/212/base 2025-12-04T09:23:39.8625326Z * [new branch] gh/fduwjj/212/head -> origin/gh/fduwjj/212/head 2025-12-04T09:23:39.8625482Z * [new branch] gh/fduwjj/212/orig -> origin/gh/fduwjj/212/orig 2025-12-04T09:23:39.8632996Z * [new branch] gh/fduwjj/213/base -> origin/gh/fduwjj/213/base 2025-12-04T09:23:39.8640557Z * [new branch] gh/fduwjj/213/head -> origin/gh/fduwjj/213/head 2025-12-04T09:23:39.8640785Z * [new branch] gh/fduwjj/213/orig -> origin/gh/fduwjj/213/orig 2025-12-04T09:23:39.8640944Z * [new branch] gh/fduwjj/226/base -> origin/gh/fduwjj/226/base 2025-12-04T09:23:39.8641097Z * [new branch] gh/fduwjj/226/head -> origin/gh/fduwjj/226/head 2025-12-04T09:23:39.8641247Z * [new branch] gh/fduwjj/226/orig -> origin/gh/fduwjj/226/orig 2025-12-04T09:23:39.8641434Z * [new branch] gh/fduwjj/229/base -> origin/gh/fduwjj/229/base 2025-12-04T09:23:39.8641577Z * [new branch] gh/fduwjj/229/head -> origin/gh/fduwjj/229/head 2025-12-04T09:23:39.8641724Z * [new branch] gh/fduwjj/229/orig -> origin/gh/fduwjj/229/orig 2025-12-04T09:23:39.8641874Z * [new branch] gh/fduwjj/233/base -> origin/gh/fduwjj/233/base 2025-12-04T09:23:39.8642007Z * [new branch] gh/fduwjj/233/head -> origin/gh/fduwjj/233/head 2025-12-04T09:23:39.8642155Z * [new branch] gh/fduwjj/233/orig -> origin/gh/fduwjj/233/orig 2025-12-04T09:23:39.8642301Z * [new branch] gh/fduwjj/234/base -> origin/gh/fduwjj/234/base 2025-12-04T09:23:39.8642449Z * [new branch] gh/fduwjj/234/head -> origin/gh/fduwjj/234/head 2025-12-04T09:23:39.8642592Z * [new branch] gh/fduwjj/234/orig -> origin/gh/fduwjj/234/orig 2025-12-04T09:23:39.8642857Z * [new branch] gh/fduwjj/235/base -> origin/gh/fduwjj/235/base 2025-12-04T09:23:39.8642993Z * [new branch] gh/fduwjj/235/head -> origin/gh/fduwjj/235/head 2025-12-04T09:23:39.8643120Z * [new branch] gh/fduwjj/235/orig -> origin/gh/fduwjj/235/orig 2025-12-04T09:23:39.8643311Z * [new branch] gh/fduwjj/236/base -> origin/gh/fduwjj/236/base 2025-12-04T09:23:39.8643450Z * [new branch] gh/fduwjj/236/head -> origin/gh/fduwjj/236/head 2025-12-04T09:23:39.8643580Z * [new branch] gh/fduwjj/236/orig -> origin/gh/fduwjj/236/orig 2025-12-04T09:23:39.8643717Z * [new branch] gh/fduwjj/237/base -> origin/gh/fduwjj/237/base 2025-12-04T09:23:39.8643845Z * [new branch] gh/fduwjj/237/head -> origin/gh/fduwjj/237/head 2025-12-04T09:23:39.8644159Z * [new branch] gh/fduwjj/237/orig -> origin/gh/fduwjj/237/orig 2025-12-04T09:23:39.8644320Z * [new branch] gh/fduwjj/238/base -> origin/gh/fduwjj/238/base 2025-12-04T09:23:39.8644468Z * [new branch] gh/fduwjj/238/head -> origin/gh/fduwjj/238/head 2025-12-04T09:23:39.8644602Z * [new branch] gh/fduwjj/238/orig -> origin/gh/fduwjj/238/orig 2025-12-04T09:23:39.8644735Z * [new branch] gh/fduwjj/239/base -> origin/gh/fduwjj/239/base 2025-12-04T09:23:39.8645083Z * [new branch] gh/fduwjj/239/head -> origin/gh/fduwjj/239/head 2025-12-04T09:23:39.8646089Z * [new branch] gh/fduwjj/239/orig -> origin/gh/fduwjj/239/orig 2025-12-04T09:23:39.8649610Z * [new branch] gh/fegin/332/base -> origin/gh/fegin/332/base 2025-12-04T09:23:39.8649780Z * [new branch] gh/fegin/332/head -> origin/gh/fegin/332/head 2025-12-04T09:23:39.8649934Z * [new branch] gh/fegin/332/orig -> origin/gh/fegin/332/orig 2025-12-04T09:23:39.8650074Z * [new branch] gh/fegin/333/base -> origin/gh/fegin/333/base 2025-12-04T09:23:39.8650356Z * [new branch] gh/fegin/333/head -> origin/gh/fegin/333/head 2025-12-04T09:23:39.8650769Z * [new branch] gh/fegin/333/orig -> origin/gh/fegin/333/orig 2025-12-04T09:23:39.8654887Z * [new branch] gh/fegin/334/base -> origin/gh/fegin/334/base 2025-12-04T09:23:39.8655212Z * [new branch] gh/fegin/334/head -> origin/gh/fegin/334/head 2025-12-04T09:23:39.8655399Z * [new branch] gh/fegin/334/orig -> origin/gh/fegin/334/orig 2025-12-04T09:23:39.8655549Z * [new branch] gh/fegin/335/base -> origin/gh/fegin/335/base 2025-12-04T09:23:39.8655689Z * [new branch] gh/fegin/335/head -> origin/gh/fegin/335/head 2025-12-04T09:23:39.8655969Z * [new branch] gh/fegin/335/orig -> origin/gh/fegin/335/orig 2025-12-04T09:23:39.8656466Z * [new branch] gh/fffrog/160/base -> origin/gh/fffrog/160/base 2025-12-04T09:23:39.8657266Z * [new branch] gh/fffrog/160/head -> origin/gh/fffrog/160/head 2025-12-04T09:23:39.8662104Z * [new branch] gh/fffrog/177/base -> origin/gh/fffrog/177/base 2025-12-04T09:23:39.8662287Z * [new branch] gh/fffrog/177/head -> origin/gh/fffrog/177/head 2025-12-04T09:23:39.8662436Z * [new branch] gh/fffrog/177/orig -> origin/gh/fffrog/177/orig 2025-12-04T09:23:39.8662575Z * [new branch] gh/fffrog/178/base -> origin/gh/fffrog/178/base 2025-12-04T09:23:39.8662707Z * [new branch] gh/fffrog/178/head -> origin/gh/fffrog/178/head 2025-12-04T09:23:39.8662839Z * [new branch] gh/fffrog/178/orig -> origin/gh/fffrog/178/orig 2025-12-04T09:23:39.8663175Z * [new branch] gh/fffrog/181/base -> origin/gh/fffrog/181/base 2025-12-04T09:23:39.8663760Z * [new branch] gh/fffrog/181/head -> origin/gh/fffrog/181/head 2025-12-04T09:23:39.8663926Z * [new branch] gh/fffrog/181/orig -> origin/gh/fffrog/181/orig 2025-12-04T09:23:39.8665247Z * [new branch] gh/fffrog/183/base -> origin/gh/fffrog/183/base 2025-12-04T09:23:39.8665494Z * [new branch] gh/fffrog/183/head -> origin/gh/fffrog/183/head 2025-12-04T09:23:39.8667513Z * [new branch] gh/fffrog/183/orig -> origin/gh/fffrog/183/orig 2025-12-04T09:23:39.8667927Z * [new branch] gh/fxdawnn/10/base -> origin/gh/fxdawnn/10/base 2025-12-04T09:23:39.8668067Z * [new branch] gh/fxdawnn/10/head -> origin/gh/fxdawnn/10/head 2025-12-04T09:23:39.8668562Z * [new branch] gh/fxdawnn/10/orig -> origin/gh/fxdawnn/10/orig 2025-12-04T09:23:39.8673018Z * [new branch] gh/fxdawnn/11/base -> origin/gh/fxdawnn/11/base 2025-12-04T09:23:39.8673190Z * [new branch] gh/fxdawnn/11/head -> origin/gh/fxdawnn/11/head 2025-12-04T09:23:39.8673331Z * [new branch] gh/fxdawnn/11/orig -> origin/gh/fxdawnn/11/orig 2025-12-04T09:23:39.8673489Z * [new branch] gh/fxdawnn/12/base -> origin/gh/fxdawnn/12/base 2025-12-04T09:23:39.8673629Z * [new branch] gh/fxdawnn/12/head -> origin/gh/fxdawnn/12/head 2025-12-04T09:23:39.8673756Z * [new branch] gh/fxdawnn/12/orig -> origin/gh/fxdawnn/12/orig 2025-12-04T09:23:39.8673914Z * [new branch] gh/fxdawnn/13/base -> origin/gh/fxdawnn/13/base 2025-12-04T09:23:39.8674736Z * [new branch] gh/fxdawnn/13/head -> origin/gh/fxdawnn/13/head 2025-12-04T09:23:39.8675302Z * [new branch] gh/fxdawnn/13/orig -> origin/gh/fxdawnn/13/orig 2025-12-04T09:23:39.8676586Z * [new branch] gh/fxdawnn/14/base -> origin/gh/fxdawnn/14/base 2025-12-04T09:23:39.8676844Z * [new branch] gh/fxdawnn/14/head -> origin/gh/fxdawnn/14/head 2025-12-04T09:23:39.8677854Z * [new branch] gh/fxdawnn/14/orig -> origin/gh/fxdawnn/14/orig 2025-12-04T09:23:39.8678493Z * [new branch] gh/fxdawnn/15/base -> origin/gh/fxdawnn/15/base 2025-12-04T09:23:39.8679216Z * [new branch] gh/fxdawnn/15/head -> origin/gh/fxdawnn/15/head 2025-12-04T09:23:39.8679810Z * [new branch] gh/fxdawnn/15/orig -> origin/gh/fxdawnn/15/orig 2025-12-04T09:23:39.8681009Z * [new branch] gh/fxdawnn/6/base -> origin/gh/fxdawnn/6/base 2025-12-04T09:23:39.8681217Z * [new branch] gh/fxdawnn/6/head -> origin/gh/fxdawnn/6/head 2025-12-04T09:23:39.8684023Z * [new branch] gh/fxdawnn/6/orig -> origin/gh/fxdawnn/6/orig 2025-12-04T09:23:39.8684364Z * [new branch] gh/fxdawnn/7/base -> origin/gh/fxdawnn/7/base 2025-12-04T09:23:39.8684516Z * [new branch] gh/fxdawnn/7/head -> origin/gh/fxdawnn/7/head 2025-12-04T09:23:39.8684724Z * [new branch] gh/fxdawnn/7/orig -> origin/gh/fxdawnn/7/orig 2025-12-04T09:23:39.8685213Z * [new branch] gh/fxdawnn/9/base -> origin/gh/fxdawnn/9/base 2025-12-04T09:23:39.8686549Z * [new branch] gh/fxdawnn/9/head -> origin/gh/fxdawnn/9/head 2025-12-04T09:23:39.8686851Z * [new branch] gh/fxdawnn/9/orig -> origin/gh/fxdawnn/9/orig 2025-12-04T09:23:39.8689050Z * [new branch] gh/galv/1/base -> origin/gh/galv/1/base 2025-12-04T09:23:39.8689362Z * [new branch] gh/galv/1/head -> origin/gh/galv/1/head 2025-12-04T09:23:39.8689503Z * [new branch] gh/galv/1/orig -> origin/gh/galv/1/orig 2025-12-04T09:23:39.8690184Z * [new branch] gh/galv/2/base -> origin/gh/galv/2/base 2025-12-04T09:23:39.8690991Z * [new branch] gh/galv/2/head -> origin/gh/galv/2/head 2025-12-04T09:23:39.8693890Z * [new branch] gh/galv/2/orig -> origin/gh/galv/2/orig 2025-12-04T09:23:39.8694321Z * [new branch] gh/galv/3/base -> origin/gh/galv/3/base 2025-12-04T09:23:39.8694458Z * [new branch] gh/galv/3/head -> origin/gh/galv/3/head 2025-12-04T09:23:39.8694583Z * [new branch] gh/galv/3/orig -> origin/gh/galv/3/orig 2025-12-04T09:23:39.8695944Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-12-04T09:23:39.8696283Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-12-04T09:23:39.8696766Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-12-04T09:23:39.8700079Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-12-04T09:23:39.8704844Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-12-04T09:23:39.8710590Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-12-04T09:23:39.8715053Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-12-04T09:23:39.8715268Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-12-04T09:23:39.8715409Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-12-04T09:23:39.8715547Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-12-04T09:23:39.8715696Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-12-04T09:23:39.8715832Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-12-04T09:23:39.8715989Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-12-04T09:23:39.8716131Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-12-04T09:23:39.8716273Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-12-04T09:23:39.8716430Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-12-04T09:23:39.8716577Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-12-04T09:23:39.8716729Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-12-04T09:23:39.8716870Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-12-04T09:23:39.8717012Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-12-04T09:23:39.8717176Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-12-04T09:23:39.8717321Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-12-04T09:23:39.8717470Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-12-04T09:23:39.8717611Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-12-04T09:23:39.8717755Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-12-04T09:23:39.8717903Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-12-04T09:23:39.8718045Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-12-04T09:23:39.8718187Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-12-04T09:23:39.8718336Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-12-04T09:23:39.8718620Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-12-04T09:23:39.8718769Z * [new branch] gh/guangyey/186/base -> origin/gh/guangyey/186/base 2025-12-04T09:23:39.8718912Z * [new branch] gh/guangyey/186/head -> origin/gh/guangyey/186/head 2025-12-04T09:23:39.8719096Z * [new branch] gh/guangyey/186/orig -> origin/gh/guangyey/186/orig 2025-12-04T09:23:39.8719245Z * [new branch] gh/guangyey/187/base -> origin/gh/guangyey/187/base 2025-12-04T09:23:39.8719391Z * [new branch] gh/guangyey/187/head -> origin/gh/guangyey/187/head 2025-12-04T09:23:39.8720050Z * [new branch] gh/guangyey/187/orig -> origin/gh/guangyey/187/orig 2025-12-04T09:23:39.8724493Z * [new branch] gh/guangyey/188/base -> origin/gh/guangyey/188/base 2025-12-04T09:23:39.8725053Z * [new branch] gh/guangyey/188/head -> origin/gh/guangyey/188/head 2025-12-04T09:23:39.8725248Z * [new branch] gh/guangyey/188/orig -> origin/gh/guangyey/188/orig 2025-12-04T09:23:39.8725407Z * [new branch] gh/guangyey/190/base -> origin/gh/guangyey/190/base 2025-12-04T09:23:39.8725559Z * [new branch] gh/guangyey/190/head -> origin/gh/guangyey/190/head 2025-12-04T09:23:39.8725715Z * [new branch] gh/guangyey/190/orig -> origin/gh/guangyey/190/orig 2025-12-04T09:23:39.8725978Z * [new branch] gh/guangyey/208/base -> origin/gh/guangyey/208/base 2025-12-04T09:23:39.8727205Z * [new branch] gh/guangyey/208/head -> origin/gh/guangyey/208/head 2025-12-04T09:23:39.8727380Z * [new branch] gh/guangyey/208/orig -> origin/gh/guangyey/208/orig 2025-12-04T09:23:39.8730231Z * [new branch] gh/guangyey/228/base -> origin/gh/guangyey/228/base 2025-12-04T09:23:39.8730570Z * [new branch] gh/guangyey/228/head -> origin/gh/guangyey/228/head 2025-12-04T09:23:39.8730758Z * [new branch] gh/guangyey/228/orig -> origin/gh/guangyey/228/orig 2025-12-04T09:23:39.8730970Z * [new branch] gh/guangyey/230/base -> origin/gh/guangyey/230/base 2025-12-04T09:23:39.8732564Z * [new branch] gh/guangyey/230/head -> origin/gh/guangyey/230/head 2025-12-04T09:23:39.8732768Z * [new branch] gh/guangyey/230/orig -> origin/gh/guangyey/230/orig 2025-12-04T09:23:39.8736435Z * [new branch] gh/guangyey/231/base -> origin/gh/guangyey/231/base 2025-12-04T09:23:39.8736625Z * [new branch] gh/guangyey/231/head -> origin/gh/guangyey/231/head 2025-12-04T09:23:39.8736771Z * [new branch] gh/guangyey/231/orig -> origin/gh/guangyey/231/orig 2025-12-04T09:23:39.8736923Z * [new branch] gh/guangyey/232/base -> origin/gh/guangyey/232/base 2025-12-04T09:23:39.8737066Z * [new branch] gh/guangyey/232/head -> origin/gh/guangyey/232/head 2025-12-04T09:23:39.8737238Z * [new branch] gh/guangyey/232/orig -> origin/gh/guangyey/232/orig 2025-12-04T09:23:39.8737641Z * [new branch] gh/guangyey/233/base -> origin/gh/guangyey/233/base 2025-12-04T09:23:39.8738968Z * [new branch] gh/guangyey/233/head -> origin/gh/guangyey/233/head 2025-12-04T09:23:39.8739252Z * [new branch] gh/guangyey/233/orig -> origin/gh/guangyey/233/orig 2025-12-04T09:23:39.8742510Z * [new branch] gh/guangyey/234/base -> origin/gh/guangyey/234/base 2025-12-04T09:23:39.8742862Z * [new branch] gh/guangyey/234/head -> origin/gh/guangyey/234/head 2025-12-04T09:23:39.8743054Z * [new branch] gh/guangyey/234/orig -> origin/gh/guangyey/234/orig 2025-12-04T09:23:39.8743225Z * [new branch] gh/guangyey/235/base -> origin/gh/guangyey/235/base 2025-12-04T09:23:39.8743650Z * [new branch] gh/guangyey/235/head -> origin/gh/guangyey/235/head 2025-12-04T09:23:39.8743818Z * [new branch] gh/guangyey/235/orig -> origin/gh/guangyey/235/orig 2025-12-04T09:23:39.8748419Z * [new branch] gh/guangyey/236/base -> origin/gh/guangyey/236/base 2025-12-04T09:23:39.8748813Z * [new branch] gh/guangyey/236/head -> origin/gh/guangyey/236/head 2025-12-04T09:23:39.8748972Z * [new branch] gh/guangyey/236/orig -> origin/gh/guangyey/236/orig 2025-12-04T09:23:39.8749110Z * [new branch] gh/guangyey/237/base -> origin/gh/guangyey/237/base 2025-12-04T09:23:39.8751759Z * [new branch] gh/guangyey/237/head -> origin/gh/guangyey/237/head 2025-12-04T09:23:39.8752115Z * [new branch] gh/guangyey/237/orig -> origin/gh/guangyey/237/orig 2025-12-04T09:23:39.8752283Z * [new branch] gh/guangyey/238/base -> origin/gh/guangyey/238/base 2025-12-04T09:23:39.8752752Z * [new branch] gh/guangyey/238/head -> origin/gh/guangyey/238/head 2025-12-04T09:23:39.8754338Z * [new branch] gh/guangyey/239/base -> origin/gh/guangyey/239/base 2025-12-04T09:23:39.8754503Z * [new branch] gh/guangyey/239/head -> origin/gh/guangyey/239/head 2025-12-04T09:23:39.8756870Z * [new branch] gh/guangyey/239/orig -> origin/gh/guangyey/239/orig 2025-12-04T09:23:39.8757209Z * [new branch] gh/guangyey/240/base -> origin/gh/guangyey/240/base 2025-12-04T09:23:39.8757786Z * [new branch] gh/guangyey/240/head -> origin/gh/guangyey/240/head 2025-12-04T09:23:39.8758974Z * [new branch] gh/guangyey/240/orig -> origin/gh/guangyey/240/orig 2025-12-04T09:23:39.8759130Z * [new branch] gh/guangyey/241/base -> origin/gh/guangyey/241/base 2025-12-04T09:23:39.8760002Z * [new branch] gh/guangyey/241/head -> origin/gh/guangyey/241/head 2025-12-04T09:23:39.8760514Z * [new branch] gh/guangyey/241/orig -> origin/gh/guangyey/241/orig 2025-12-04T09:23:39.8761829Z * [new branch] gh/guangyey/242/base -> origin/gh/guangyey/242/base 2025-12-04T09:23:39.8768711Z * [new branch] gh/guangyey/242/head -> origin/gh/guangyey/242/head 2025-12-04T09:23:39.8769071Z * [new branch] gh/guangyey/242/orig -> origin/gh/guangyey/242/orig 2025-12-04T09:23:39.8769221Z * [new branch] gh/guangyey/243/base -> origin/gh/guangyey/243/base 2025-12-04T09:23:39.8769356Z * [new branch] gh/guangyey/243/head -> origin/gh/guangyey/243/head 2025-12-04T09:23:39.8769490Z * [new branch] gh/guangyey/243/orig -> origin/gh/guangyey/243/orig 2025-12-04T09:23:39.8769634Z * [new branch] gh/guangyey/244/base -> origin/gh/guangyey/244/base 2025-12-04T09:23:39.8769767Z * [new branch] gh/guangyey/244/head -> origin/gh/guangyey/244/head 2025-12-04T09:23:39.8769921Z * [new branch] gh/guangyey/244/orig -> origin/gh/guangyey/244/orig 2025-12-04T09:23:39.8773347Z * [new branch] gh/guangyey/245/base -> origin/gh/guangyey/245/base 2025-12-04T09:23:39.8773694Z * [new branch] gh/guangyey/245/head -> origin/gh/guangyey/245/head 2025-12-04T09:23:39.8773919Z * [new branch] gh/guangyey/245/orig -> origin/gh/guangyey/245/orig 2025-12-04T09:23:39.8774153Z * [new branch] gh/guangyey/246/base -> origin/gh/guangyey/246/base 2025-12-04T09:23:39.8774320Z * [new branch] gh/guangyey/246/head -> origin/gh/guangyey/246/head 2025-12-04T09:23:39.8774474Z * [new branch] gh/guangyey/246/orig -> origin/gh/guangyey/246/orig 2025-12-04T09:23:39.8774765Z * [new branch] gh/guangyey/247/base -> origin/gh/guangyey/247/base 2025-12-04T09:23:39.8778658Z * [new branch] gh/guangyey/247/head -> origin/gh/guangyey/247/head 2025-12-04T09:23:39.8778995Z * [new branch] gh/guangyey/247/orig -> origin/gh/guangyey/247/orig 2025-12-04T09:23:39.8779185Z * [new branch] gh/guangyey/248/base -> origin/gh/guangyey/248/base 2025-12-04T09:23:39.8779489Z * [new branch] gh/guangyey/248/head -> origin/gh/guangyey/248/head 2025-12-04T09:23:39.8779766Z * [new branch] gh/guangyey/248/orig -> origin/gh/guangyey/248/orig 2025-12-04T09:23:39.8779926Z * [new branch] gh/guangyey/249/base -> origin/gh/guangyey/249/base 2025-12-04T09:23:39.8780161Z * [new branch] gh/guangyey/249/head -> origin/gh/guangyey/249/head 2025-12-04T09:23:39.8780298Z * [new branch] gh/guangyey/249/orig -> origin/gh/guangyey/249/orig 2025-12-04T09:23:39.8780449Z * [new branch] gh/guangyey/250/base -> origin/gh/guangyey/250/base 2025-12-04T09:23:39.8780594Z * [new branch] gh/guangyey/250/head -> origin/gh/guangyey/250/head 2025-12-04T09:23:39.8780727Z * [new branch] gh/guangyey/250/orig -> origin/gh/guangyey/250/orig 2025-12-04T09:23:39.8780867Z * [new branch] gh/guangyey/251/base -> origin/gh/guangyey/251/base 2025-12-04T09:23:39.8781004Z * [new branch] gh/guangyey/251/head -> origin/gh/guangyey/251/head 2025-12-04T09:23:39.8788236Z * [new branch] gh/guangyey/251/orig -> origin/gh/guangyey/251/orig 2025-12-04T09:23:39.8788580Z * [new branch] gh/guangyey/252/base -> origin/gh/guangyey/252/base 2025-12-04T09:23:39.8788804Z * [new branch] gh/guangyey/252/head -> origin/gh/guangyey/252/head 2025-12-04T09:23:39.8788991Z * [new branch] gh/guangyey/252/orig -> origin/gh/guangyey/252/orig 2025-12-04T09:23:39.8789167Z * [new branch] gh/guangyey/253/base -> origin/gh/guangyey/253/base 2025-12-04T09:23:39.8789331Z * [new branch] gh/guangyey/253/head -> origin/gh/guangyey/253/head 2025-12-04T09:23:39.8789613Z * [new branch] gh/guangyey/253/orig -> origin/gh/guangyey/253/orig 2025-12-04T09:23:39.8789775Z * [new branch] gh/guangyey/254/base -> origin/gh/guangyey/254/base 2025-12-04T09:23:39.8790037Z * [new branch] gh/guangyey/254/head -> origin/gh/guangyey/254/head 2025-12-04T09:23:39.8790205Z * [new branch] gh/guangyey/254/orig -> origin/gh/guangyey/254/orig 2025-12-04T09:23:39.8790356Z * [new branch] gh/guangyey/255/base -> origin/gh/guangyey/255/base 2025-12-04T09:23:39.8790509Z * [new branch] gh/guangyey/255/head -> origin/gh/guangyey/255/head 2025-12-04T09:23:39.8790657Z * [new branch] gh/guangyey/255/orig -> origin/gh/guangyey/255/orig 2025-12-04T09:23:39.8790871Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-12-04T09:23:39.8795285Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-12-04T09:23:39.8795640Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-12-04T09:23:39.8795868Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-12-04T09:23:39.8796133Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-12-04T09:23:39.8796328Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-12-04T09:23:39.8796985Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-12-04T09:23:39.8797187Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-12-04T09:23:39.8797885Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-12-04T09:23:39.8798979Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-12-04T09:23:39.8799548Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-12-04T09:23:39.8804718Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-12-04T09:23:39.8805230Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-12-04T09:23:39.8805565Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-12-04T09:23:39.8805781Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-12-04T09:23:39.8806072Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-12-04T09:23:39.8806264Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-12-04T09:23:39.8806561Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-12-04T09:23:39.8806743Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-12-04T09:23:39.8807364Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-12-04T09:23:39.8807614Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-12-04T09:23:39.8811209Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-12-04T09:23:39.8811432Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-12-04T09:23:39.8811604Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-12-04T09:23:39.8811777Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-12-04T09:23:39.8811963Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-12-04T09:23:39.8812550Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-12-04T09:23:39.8816307Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-12-04T09:23:39.8816533Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-12-04T09:23:39.8816720Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-12-04T09:23:39.8816934Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-12-04T09:23:39.8817642Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-12-04T09:23:39.8817843Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-12-04T09:23:39.8819492Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-12-04T09:23:39.8819758Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-12-04T09:23:39.8819979Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-12-04T09:23:39.8822600Z * [new branch] gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base 2025-12-04T09:23:39.8822815Z * [new branch] gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head 2025-12-04T09:23:39.8822988Z * [new branch] gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig 2025-12-04T09:23:39.8823171Z * [new branch] gh/guilhermeleobas/247/base -> origin/gh/guilhermeleobas/247/base 2025-12-04T09:23:39.8823714Z * [new branch] gh/guilhermeleobas/247/head -> origin/gh/guilhermeleobas/247/head 2025-12-04T09:23:39.8824647Z * [new branch] gh/guilhermeleobas/247/orig -> origin/gh/guilhermeleobas/247/orig 2025-12-04T09:23:39.8825349Z * [new branch] gh/guilhermeleobas/248/base -> origin/gh/guilhermeleobas/248/base 2025-12-04T09:23:39.8826152Z * [new branch] gh/guilhermeleobas/248/head -> origin/gh/guilhermeleobas/248/head 2025-12-04T09:23:39.8830631Z * [new branch] gh/guilhermeleobas/248/orig -> origin/gh/guilhermeleobas/248/orig 2025-12-04T09:23:39.8830986Z * [new branch] gh/guilhermeleobas/250/base -> origin/gh/guilhermeleobas/250/base 2025-12-04T09:23:39.8831173Z * [new branch] gh/guilhermeleobas/250/head -> origin/gh/guilhermeleobas/250/head 2025-12-04T09:23:39.8831345Z * [new branch] gh/guilhermeleobas/250/orig -> origin/gh/guilhermeleobas/250/orig 2025-12-04T09:23:39.8831521Z * [new branch] gh/guilhermeleobas/253/base -> origin/gh/guilhermeleobas/253/base 2025-12-04T09:23:39.8831694Z * [new branch] gh/guilhermeleobas/253/head -> origin/gh/guilhermeleobas/253/head 2025-12-04T09:23:39.8832638Z * [new branch] gh/guilhermeleobas/253/orig -> origin/gh/guilhermeleobas/253/orig 2025-12-04T09:23:39.8836547Z * [new branch] gh/guilhermeleobas/254/base -> origin/gh/guilhermeleobas/254/base 2025-12-04T09:23:39.8836782Z * [new branch] gh/guilhermeleobas/254/head -> origin/gh/guilhermeleobas/254/head 2025-12-04T09:23:39.8836981Z * [new branch] gh/guilhermeleobas/254/orig -> origin/gh/guilhermeleobas/254/orig 2025-12-04T09:23:39.8837155Z * [new branch] gh/guilhermeleobas/255/base -> origin/gh/guilhermeleobas/255/base 2025-12-04T09:23:39.8837333Z * [new branch] gh/guilhermeleobas/255/head -> origin/gh/guilhermeleobas/255/head 2025-12-04T09:23:39.8837499Z * [new branch] gh/guilhermeleobas/255/orig -> origin/gh/guilhermeleobas/255/orig 2025-12-04T09:23:39.8837671Z * [new branch] gh/guilhermeleobas/256/base -> origin/gh/guilhermeleobas/256/base 2025-12-04T09:23:39.8838209Z * [new branch] gh/guilhermeleobas/256/head -> origin/gh/guilhermeleobas/256/head 2025-12-04T09:23:39.8844850Z * [new branch] gh/guilhermeleobas/256/orig -> origin/gh/guilhermeleobas/256/orig 2025-12-04T09:23:39.8847853Z * [new branch] gh/guilhermeleobas/257/base -> origin/gh/guilhermeleobas/257/base 2025-12-04T09:23:39.8848198Z * [new branch] gh/guilhermeleobas/257/head -> origin/gh/guilhermeleobas/257/head 2025-12-04T09:23:39.8853077Z * [new branch] gh/guilhermeleobas/257/orig -> origin/gh/guilhermeleobas/257/orig 2025-12-04T09:23:39.8854957Z * [new branch] gh/guilhermeleobas/258/base -> origin/gh/guilhermeleobas/258/base 2025-12-04T09:23:39.8855163Z * [new branch] gh/guilhermeleobas/258/head -> origin/gh/guilhermeleobas/258/head 2025-12-04T09:23:39.8855388Z * [new branch] gh/guilhermeleobas/258/orig -> origin/gh/guilhermeleobas/258/orig 2025-12-04T09:23:39.8855571Z * [new branch] gh/guilhermeleobas/259/base -> origin/gh/guilhermeleobas/259/base 2025-12-04T09:23:39.8855772Z * [new branch] gh/guilhermeleobas/259/head -> origin/gh/guilhermeleobas/259/head 2025-12-04T09:23:39.8855952Z * [new branch] gh/guilhermeleobas/259/orig -> origin/gh/guilhermeleobas/259/orig 2025-12-04T09:23:39.8856131Z * [new branch] gh/guilhermeleobas/260/base -> origin/gh/guilhermeleobas/260/base 2025-12-04T09:23:39.8856323Z * [new branch] gh/guilhermeleobas/260/head -> origin/gh/guilhermeleobas/260/head 2025-12-04T09:23:39.8856506Z * [new branch] gh/guilhermeleobas/260/orig -> origin/gh/guilhermeleobas/260/orig 2025-12-04T09:23:39.8856686Z * [new branch] gh/guilhermeleobas/261/base -> origin/gh/guilhermeleobas/261/base 2025-12-04T09:23:39.8856862Z * [new branch] gh/guilhermeleobas/261/head -> origin/gh/guilhermeleobas/261/head 2025-12-04T09:23:39.8857038Z * [new branch] gh/guilhermeleobas/261/orig -> origin/gh/guilhermeleobas/261/orig 2025-12-04T09:23:39.8857585Z * [new branch] gh/guilhermeleobas/262/base -> origin/gh/guilhermeleobas/262/base 2025-12-04T09:23:39.8862295Z * [new branch] gh/guilhermeleobas/262/head -> origin/gh/guilhermeleobas/262/head 2025-12-04T09:23:39.8867198Z * [new branch] gh/guilhermeleobas/262/orig -> origin/gh/guilhermeleobas/262/orig 2025-12-04T09:23:39.8869641Z * [new branch] gh/guilhermeleobas/263/base -> origin/gh/guilhermeleobas/263/base 2025-12-04T09:23:39.8870050Z * [new branch] gh/guilhermeleobas/263/head -> origin/gh/guilhermeleobas/263/head 2025-12-04T09:23:39.8870261Z * [new branch] gh/guilhermeleobas/263/orig -> origin/gh/guilhermeleobas/263/orig 2025-12-04T09:23:39.8870515Z * [new branch] gh/guilhermeleobas/264/base -> origin/gh/guilhermeleobas/264/base 2025-12-04T09:23:39.8870725Z * [new branch] gh/guilhermeleobas/264/head -> origin/gh/guilhermeleobas/264/head 2025-12-04T09:23:39.8870904Z * [new branch] gh/guilhermeleobas/264/orig -> origin/gh/guilhermeleobas/264/orig 2025-12-04T09:23:39.8871216Z * [new branch] gh/guilhermeleobas/265/base -> origin/gh/guilhermeleobas/265/base 2025-12-04T09:23:39.8871389Z * [new branch] gh/guilhermeleobas/265/head -> origin/gh/guilhermeleobas/265/head 2025-12-04T09:23:39.8871706Z * [new branch] gh/guilhermeleobas/265/orig -> origin/gh/guilhermeleobas/265/orig 2025-12-04T09:23:39.8872394Z * [new branch] gh/guilhermeleobas/266/base -> origin/gh/guilhermeleobas/266/base 2025-12-04T09:23:39.8872616Z * [new branch] gh/guilhermeleobas/266/head -> origin/gh/guilhermeleobas/266/head 2025-12-04T09:23:39.8872786Z * [new branch] gh/guilhermeleobas/266/orig -> origin/gh/guilhermeleobas/266/orig 2025-12-04T09:23:39.8872953Z * [new branch] gh/guilhermeleobas/267/base -> origin/gh/guilhermeleobas/267/base 2025-12-04T09:23:39.8873125Z * [new branch] gh/guilhermeleobas/267/head -> origin/gh/guilhermeleobas/267/head 2025-12-04T09:23:39.8873304Z * [new branch] gh/guilhermeleobas/267/orig -> origin/gh/guilhermeleobas/267/orig 2025-12-04T09:23:39.8873482Z * [new branch] gh/hameerabbasi/1/base -> origin/gh/hameerabbasi/1/base 2025-12-04T09:23:39.8873634Z * [new branch] gh/hameerabbasi/1/head -> origin/gh/hameerabbasi/1/head 2025-12-04T09:23:39.8873778Z * [new branch] gh/hameerabbasi/2/base -> origin/gh/hameerabbasi/2/base 2025-12-04T09:23:39.8873930Z * [new branch] gh/hameerabbasi/2/head -> origin/gh/hameerabbasi/2/head 2025-12-04T09:23:39.8874073Z * [new branch] gh/hameerabbasi/2/orig -> origin/gh/hameerabbasi/2/orig 2025-12-04T09:23:39.8874221Z * [new branch] gh/hameerabbasi/3/base -> origin/gh/hameerabbasi/3/base 2025-12-04T09:23:39.8874363Z * [new branch] gh/hameerabbasi/3/head -> origin/gh/hameerabbasi/3/head 2025-12-04T09:23:39.8874509Z * [new branch] gh/hameerabbasi/3/orig -> origin/gh/hameerabbasi/3/orig 2025-12-04T09:23:39.8874655Z * [new branch] gh/hameerabbasi/4/base -> origin/gh/hameerabbasi/4/base 2025-12-04T09:23:39.8876029Z * [new branch] gh/hameerabbasi/4/head -> origin/gh/hameerabbasi/4/head 2025-12-04T09:23:39.8876638Z * [new branch] gh/hameerabbasi/4/orig -> origin/gh/hameerabbasi/4/orig 2025-12-04T09:23:39.8876827Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-12-04T09:23:39.8876979Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-12-04T09:23:39.8877131Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-12-04T09:23:39.8884312Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-12-04T09:23:39.8884943Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-12-04T09:23:39.8885282Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-12-04T09:23:39.8885442Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-12-04T09:23:39.8885583Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-12-04T09:23:39.8885818Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-12-04T09:23:39.8891187Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-12-04T09:23:39.8893327Z * [new branch] gh/isuruf/146/base -> origin/gh/isuruf/146/base 2025-12-04T09:23:39.8893652Z * [new branch] gh/isuruf/146/head -> origin/gh/isuruf/146/head 2025-12-04T09:23:39.8893806Z * [new branch] gh/isuruf/146/orig -> origin/gh/isuruf/146/orig 2025-12-04T09:23:39.8894053Z * [new branch] gh/isuruf/158/base -> origin/gh/isuruf/158/base 2025-12-04T09:23:39.8894238Z * [new branch] gh/isuruf/158/head -> origin/gh/isuruf/158/head 2025-12-04T09:23:39.8894392Z * [new branch] gh/isuruf/159/base -> origin/gh/isuruf/159/base 2025-12-04T09:23:39.8894615Z * [new branch] gh/isuruf/159/head -> origin/gh/isuruf/159/head 2025-12-04T09:23:39.8894796Z * [new branch] gh/isuruf/160/base -> origin/gh/isuruf/160/base 2025-12-04T09:23:39.8895050Z * [new branch] gh/isuruf/160/head -> origin/gh/isuruf/160/head 2025-12-04T09:23:39.8895196Z * [new branch] gh/isuruf/160/orig -> origin/gh/isuruf/160/orig 2025-12-04T09:23:39.8895431Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-12-04T09:23:39.8895589Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-12-04T09:23:39.8895813Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-12-04T09:23:39.8899894Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-12-04T09:23:39.8900251Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-12-04T09:23:39.8900438Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-12-04T09:23:39.8900631Z * [new branch] gh/jamesjwu/187/base -> origin/gh/jamesjwu/187/base 2025-12-04T09:23:39.8900811Z * [new branch] gh/jamesjwu/187/head -> origin/gh/jamesjwu/187/head 2025-12-04T09:23:39.8901002Z * [new branch] gh/jamesjwu/187/orig -> origin/gh/jamesjwu/187/orig 2025-12-04T09:23:39.8901666Z * [new branch] gh/jamesjwu/196/base -> origin/gh/jamesjwu/196/base 2025-12-04T09:23:39.8901863Z * [new branch] gh/jamesjwu/196/head -> origin/gh/jamesjwu/196/head 2025-12-04T09:23:39.8902010Z * [new branch] gh/jamesjwu/196/orig -> origin/gh/jamesjwu/196/orig 2025-12-04T09:23:39.8902171Z * [new branch] gh/jamesjwu/198/base -> origin/gh/jamesjwu/198/base 2025-12-04T09:23:39.8902317Z * [new branch] gh/jamesjwu/198/head -> origin/gh/jamesjwu/198/head 2025-12-04T09:23:39.8902456Z * [new branch] gh/jamesjwu/198/orig -> origin/gh/jamesjwu/198/orig 2025-12-04T09:23:39.8902608Z * [new branch] gh/jamesjwu/207/base -> origin/gh/jamesjwu/207/base 2025-12-04T09:23:39.8902758Z * [new branch] gh/jamesjwu/207/head -> origin/gh/jamesjwu/207/head 2025-12-04T09:23:39.8902897Z * [new branch] gh/jamesjwu/207/orig -> origin/gh/jamesjwu/207/orig 2025-12-04T09:23:39.8909718Z * [new branch] gh/jamesjwu/208/base -> origin/gh/jamesjwu/208/base 2025-12-04T09:23:39.8914901Z * [new branch] gh/jamesjwu/208/head -> origin/gh/jamesjwu/208/head 2025-12-04T09:23:39.8917782Z * [new branch] gh/jamesjwu/208/orig -> origin/gh/jamesjwu/208/orig 2025-12-04T09:23:39.8918333Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-12-04T09:23:39.8918527Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-12-04T09:23:39.8918829Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-12-04T09:23:39.8918987Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-12-04T09:23:39.8919145Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-12-04T09:23:39.8919302Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-12-04T09:23:39.8919456Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-12-04T09:23:39.8919603Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-12-04T09:23:39.8919757Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-12-04T09:23:39.8919904Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-12-04T09:23:39.8920052Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-12-04T09:23:39.8920209Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-12-04T09:23:39.8920562Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-12-04T09:23:39.8920723Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-12-04T09:23:39.8920877Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-12-04T09:23:39.8921023Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-12-04T09:23:39.8921174Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-12-04T09:23:39.8921336Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-12-04T09:23:39.8921484Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-12-04T09:23:39.8921639Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-12-04T09:23:39.8921791Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-12-04T09:23:39.8921948Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-12-04T09:23:39.8922106Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-12-04T09:23:39.8922256Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-12-04T09:23:39.8922579Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-12-04T09:23:39.8929446Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-12-04T09:23:39.8929656Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-12-04T09:23:39.8929807Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-12-04T09:23:39.8929976Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-12-04T09:23:39.8930155Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-12-04T09:23:39.8930309Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-12-04T09:23:39.8930983Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-12-04T09:23:39.8931284Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-12-04T09:23:39.8931438Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-12-04T09:23:39.8931594Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-12-04T09:23:39.8931810Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-12-04T09:23:39.8931959Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-12-04T09:23:39.8935984Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-12-04T09:23:39.8936223Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-12-04T09:23:39.8936361Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-12-04T09:23:39.8936645Z * [new branch] gh/janeyx99/302/base -> origin/gh/janeyx99/302/base 2025-12-04T09:23:39.8936811Z * [new branch] gh/janeyx99/302/head -> origin/gh/janeyx99/302/head 2025-12-04T09:23:39.8936957Z * [new branch] gh/janeyx99/303/base -> origin/gh/janeyx99/303/base 2025-12-04T09:23:39.8945047Z * [new branch] gh/janeyx99/303/head -> origin/gh/janeyx99/303/head 2025-12-04T09:23:39.8945255Z * [new branch] gh/janeyx99/305/base -> origin/gh/janeyx99/305/base 2025-12-04T09:23:39.8945833Z * [new branch] gh/janeyx99/305/head -> origin/gh/janeyx99/305/head 2025-12-04T09:23:39.8946021Z * [new branch] gh/janeyx99/306/base -> origin/gh/janeyx99/306/base 2025-12-04T09:23:39.8946178Z * [new branch] gh/janeyx99/306/head -> origin/gh/janeyx99/306/head 2025-12-04T09:23:39.8946343Z * [new branch] gh/janeyx99/314/base -> origin/gh/janeyx99/314/base 2025-12-04T09:23:39.8946496Z * [new branch] gh/janeyx99/314/head -> origin/gh/janeyx99/314/head 2025-12-04T09:23:39.8946643Z * [new branch] gh/janeyx99/314/orig -> origin/gh/janeyx99/314/orig 2025-12-04T09:23:39.8946791Z * [new branch] gh/janeyx99/315/base -> origin/gh/janeyx99/315/base 2025-12-04T09:23:39.8946945Z * [new branch] gh/janeyx99/315/head -> origin/gh/janeyx99/315/head 2025-12-04T09:23:39.8947367Z * [new branch] gh/janeyx99/315/orig -> origin/gh/janeyx99/315/orig 2025-12-04T09:23:39.8947522Z * [new branch] gh/janeyx99/316/base -> origin/gh/janeyx99/316/base 2025-12-04T09:23:39.8947685Z * [new branch] gh/janeyx99/316/head -> origin/gh/janeyx99/316/head 2025-12-04T09:23:39.8949808Z * [new branch] gh/janeyx99/316/orig -> origin/gh/janeyx99/316/orig 2025-12-04T09:23:39.8949985Z * [new branch] gh/janeyx99/317/base -> origin/gh/janeyx99/317/base 2025-12-04T09:23:39.8950135Z * [new branch] gh/janeyx99/317/head -> origin/gh/janeyx99/317/head 2025-12-04T09:23:39.8950417Z * [new branch] gh/janeyx99/317/orig -> origin/gh/janeyx99/317/orig 2025-12-04T09:23:39.8951863Z * [new branch] gh/janeyx99/325/base -> origin/gh/janeyx99/325/base 2025-12-04T09:23:39.8952072Z * [new branch] gh/janeyx99/325/head -> origin/gh/janeyx99/325/head 2025-12-04T09:23:39.8952234Z * [new branch] gh/janeyx99/325/orig -> origin/gh/janeyx99/325/orig 2025-12-04T09:23:39.8952385Z * [new branch] gh/janeyx99/327/base -> origin/gh/janeyx99/327/base 2025-12-04T09:23:39.8952938Z * [new branch] gh/janeyx99/327/head -> origin/gh/janeyx99/327/head 2025-12-04T09:23:39.8953194Z * [new branch] gh/janeyx99/327/orig -> origin/gh/janeyx99/327/orig 2025-12-04T09:23:39.8954514Z * [new branch] gh/janeyx99/328/base -> origin/gh/janeyx99/328/base 2025-12-04T09:23:39.8954888Z * [new branch] gh/janeyx99/328/head -> origin/gh/janeyx99/328/head 2025-12-04T09:23:39.8956001Z * [new branch] gh/janeyx99/328/orig -> origin/gh/janeyx99/328/orig 2025-12-04T09:23:39.8956680Z * [new branch] gh/janeyx99/329/base -> origin/gh/janeyx99/329/base 2025-12-04T09:23:39.8957305Z * [new branch] gh/janeyx99/329/head -> origin/gh/janeyx99/329/head 2025-12-04T09:23:39.8958257Z * [new branch] gh/janeyx99/329/orig -> origin/gh/janeyx99/329/orig 2025-12-04T09:23:39.8959634Z * [new branch] gh/janeyx99/330/base -> origin/gh/janeyx99/330/base 2025-12-04T09:23:39.8960299Z * [new branch] gh/janeyx99/330/head -> origin/gh/janeyx99/330/head 2025-12-04T09:23:39.8961146Z * [new branch] gh/janeyx99/330/orig -> origin/gh/janeyx99/330/orig 2025-12-04T09:23:39.8962297Z * [new branch] gh/janeyx99/331/base -> origin/gh/janeyx99/331/base 2025-12-04T09:23:39.8962918Z * [new branch] gh/janeyx99/331/head -> origin/gh/janeyx99/331/head 2025-12-04T09:23:39.8963282Z * [new branch] gh/janeyx99/331/orig -> origin/gh/janeyx99/331/orig 2025-12-04T09:23:39.8964871Z * [new branch] gh/janeyx99/332/base -> origin/gh/janeyx99/332/base 2025-12-04T09:23:39.8965062Z * [new branch] gh/janeyx99/332/head -> origin/gh/janeyx99/332/head 2025-12-04T09:23:39.8965940Z * [new branch] gh/janeyx99/332/orig -> origin/gh/janeyx99/332/orig 2025-12-04T09:23:39.8970265Z * [new branch] gh/janeyx99/333/base -> origin/gh/janeyx99/333/base 2025-12-04T09:23:39.8970444Z * [new branch] gh/janeyx99/333/head -> origin/gh/janeyx99/333/head 2025-12-04T09:23:39.8970601Z * [new branch] gh/janeyx99/333/orig -> origin/gh/janeyx99/333/orig 2025-12-04T09:23:39.8970753Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-12-04T09:23:39.8970894Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-12-04T09:23:39.8971045Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-12-04T09:23:39.8973873Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-12-04T09:23:39.8979463Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-12-04T09:23:39.8982537Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-12-04T09:23:39.8987503Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-12-04T09:23:39.8989579Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-12-04T09:23:39.8989775Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-12-04T09:23:39.8989918Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-12-04T09:23:39.8990069Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-12-04T09:23:39.8990215Z * [new branch] gh/jansel/533/base -> origin/gh/jansel/533/base 2025-12-04T09:23:39.8990372Z * [new branch] gh/jansel/533/head -> origin/gh/jansel/533/head 2025-12-04T09:23:39.8990520Z * [new branch] gh/jansel/533/orig -> origin/gh/jansel/533/orig 2025-12-04T09:23:39.8990664Z * [new branch] gh/jansel/552/base -> origin/gh/jansel/552/base 2025-12-04T09:23:39.8990816Z * [new branch] gh/jansel/552/head -> origin/gh/jansel/552/head 2025-12-04T09:23:39.8990952Z * [new branch] gh/jansel/552/orig -> origin/gh/jansel/552/orig 2025-12-04T09:23:39.8991095Z * [new branch] gh/jansel/553/base -> origin/gh/jansel/553/base 2025-12-04T09:23:39.8991242Z * [new branch] gh/jansel/553/head -> origin/gh/jansel/553/head 2025-12-04T09:23:39.8991385Z * [new branch] gh/jansel/553/orig -> origin/gh/jansel/553/orig 2025-12-04T09:23:39.8991525Z * [new branch] gh/jansel/554/base -> origin/gh/jansel/554/base 2025-12-04T09:23:39.8991838Z * [new branch] gh/jansel/554/head -> origin/gh/jansel/554/head 2025-12-04T09:23:39.8991984Z * [new branch] gh/jansel/554/orig -> origin/gh/jansel/554/orig 2025-12-04T09:23:39.8992132Z * [new branch] gh/jansel/555/base -> origin/gh/jansel/555/base 2025-12-04T09:23:39.8992319Z * [new branch] gh/jansel/555/head -> origin/gh/jansel/555/head 2025-12-04T09:23:39.8992635Z * [new branch] gh/jansel/555/orig -> origin/gh/jansel/555/orig 2025-12-04T09:23:39.8992809Z * [new branch] gh/jansel/556/base -> origin/gh/jansel/556/base 2025-12-04T09:23:39.8992954Z * [new branch] gh/jansel/556/head -> origin/gh/jansel/556/head 2025-12-04T09:23:39.8993099Z * [new branch] gh/jansel/556/orig -> origin/gh/jansel/556/orig 2025-12-04T09:23:39.8995239Z * [new branch] gh/jansel/557/base -> origin/gh/jansel/557/base 2025-12-04T09:23:39.8995576Z * [new branch] gh/jansel/557/head -> origin/gh/jansel/557/head 2025-12-04T09:23:39.8995724Z * [new branch] gh/jansel/557/orig -> origin/gh/jansel/557/orig 2025-12-04T09:23:39.8996003Z * [new branch] gh/jansel/558/base -> origin/gh/jansel/558/base 2025-12-04T09:23:39.8998989Z * [new branch] gh/jansel/558/head -> origin/gh/jansel/558/head 2025-12-04T09:23:39.8999169Z * [new branch] gh/jansel/558/orig -> origin/gh/jansel/558/orig 2025-12-04T09:23:39.8999332Z * [new branch] gh/jansel/559/base -> origin/gh/jansel/559/base 2025-12-04T09:23:39.8999485Z * [new branch] gh/jansel/559/head -> origin/gh/jansel/559/head 2025-12-04T09:23:39.8999938Z * [new branch] gh/jansel/559/orig -> origin/gh/jansel/559/orig 2025-12-04T09:23:39.9001227Z * [new branch] gh/jansel/560/base -> origin/gh/jansel/560/base 2025-12-04T09:23:39.9001403Z * [new branch] gh/jansel/560/head -> origin/gh/jansel/560/head 2025-12-04T09:23:39.9004728Z * [new branch] gh/jansel/560/orig -> origin/gh/jansel/560/orig 2025-12-04T09:23:39.9009626Z * [new branch] gh/jansel/561/base -> origin/gh/jansel/561/base 2025-12-04T09:23:39.9014601Z * [new branch] gh/jansel/561/head -> origin/gh/jansel/561/head 2025-12-04T09:23:39.9020267Z * [new branch] gh/jansel/561/orig -> origin/gh/jansel/561/orig 2025-12-04T09:23:39.9020573Z * [new branch] gh/jansel/562/base -> origin/gh/jansel/562/base 2025-12-04T09:23:39.9021038Z * [new branch] gh/jansel/562/head -> origin/gh/jansel/562/head 2025-12-04T09:23:39.9021223Z * [new branch] gh/jansel/562/orig -> origin/gh/jansel/562/orig 2025-12-04T09:23:39.9021381Z * [new branch] gh/jansel/563/base -> origin/gh/jansel/563/base 2025-12-04T09:23:39.9021541Z * [new branch] gh/jansel/563/head -> origin/gh/jansel/563/head 2025-12-04T09:23:39.9021693Z * [new branch] gh/jansel/563/orig -> origin/gh/jansel/563/orig 2025-12-04T09:23:39.9021839Z * [new branch] gh/jansel/564/base -> origin/gh/jansel/564/base 2025-12-04T09:23:39.9021992Z * [new branch] gh/jansel/564/head -> origin/gh/jansel/564/head 2025-12-04T09:23:39.9022138Z * [new branch] gh/jansel/564/orig -> origin/gh/jansel/564/orig 2025-12-04T09:23:39.9022283Z * [new branch] gh/jansel/565/base -> origin/gh/jansel/565/base 2025-12-04T09:23:39.9022432Z * [new branch] gh/jansel/565/head -> origin/gh/jansel/565/head 2025-12-04T09:23:39.9022576Z * [new branch] gh/jansel/565/orig -> origin/gh/jansel/565/orig 2025-12-04T09:23:39.9022889Z * [new branch] gh/jansel/566/base -> origin/gh/jansel/566/base 2025-12-04T09:23:39.9023034Z * [new branch] gh/jansel/566/head -> origin/gh/jansel/566/head 2025-12-04T09:23:39.9023171Z * [new branch] gh/jansel/566/orig -> origin/gh/jansel/566/orig 2025-12-04T09:23:39.9023310Z * [new branch] gh/jansel/567/base -> origin/gh/jansel/567/base 2025-12-04T09:23:39.9023508Z * [new branch] gh/jansel/567/head -> origin/gh/jansel/567/head 2025-12-04T09:23:39.9023657Z * [new branch] gh/jansel/567/orig -> origin/gh/jansel/567/orig 2025-12-04T09:23:39.9023800Z * [new branch] gh/jansel/568/base -> origin/gh/jansel/568/base 2025-12-04T09:23:39.9023934Z * [new branch] gh/jansel/568/head -> origin/gh/jansel/568/head 2025-12-04T09:23:39.9024077Z * [new branch] gh/jansel/568/orig -> origin/gh/jansel/568/orig 2025-12-04T09:23:39.9024227Z * [new branch] gh/jansel/569/base -> origin/gh/jansel/569/base 2025-12-04T09:23:39.9024374Z * [new branch] gh/jansel/569/head -> origin/gh/jansel/569/head 2025-12-04T09:23:39.9024570Z * [new branch] gh/jansel/569/orig -> origin/gh/jansel/569/orig 2025-12-04T09:23:39.9026832Z * [new branch] gh/jansel/570/base -> origin/gh/jansel/570/base 2025-12-04T09:23:39.9027015Z * [new branch] gh/jansel/570/head -> origin/gh/jansel/570/head 2025-12-04T09:23:39.9027535Z * [new branch] gh/jansel/570/orig -> origin/gh/jansel/570/orig 2025-12-04T09:23:39.9027753Z * [new branch] gh/jansel/571/base -> origin/gh/jansel/571/base 2025-12-04T09:23:39.9028244Z * [new branch] gh/jansel/571/head -> origin/gh/jansel/571/head 2025-12-04T09:23:39.9029310Z * [new branch] gh/jansel/571/orig -> origin/gh/jansel/571/orig 2025-12-04T09:23:39.9030079Z * [new branch] gh/jansel/572/base -> origin/gh/jansel/572/base 2025-12-04T09:23:39.9030708Z * [new branch] gh/jansel/572/head -> origin/gh/jansel/572/head 2025-12-04T09:23:39.9031599Z * [new branch] gh/jansel/572/orig -> origin/gh/jansel/572/orig 2025-12-04T09:23:39.9036240Z * [new branch] gh/jansel/573/base -> origin/gh/jansel/573/base 2025-12-04T09:23:39.9036426Z * [new branch] gh/jansel/573/head -> origin/gh/jansel/573/head 2025-12-04T09:23:39.9036582Z * [new branch] gh/jansel/573/orig -> origin/gh/jansel/573/orig 2025-12-04T09:23:39.9036742Z * [new branch] gh/jansel/574/base -> origin/gh/jansel/574/base 2025-12-04T09:23:39.9036892Z * [new branch] gh/jansel/574/head -> origin/gh/jansel/574/head 2025-12-04T09:23:39.9037045Z * [new branch] gh/jansel/574/orig -> origin/gh/jansel/574/orig 2025-12-04T09:23:39.9037576Z * [new branch] gh/jansel/575/base -> origin/gh/jansel/575/base 2025-12-04T09:23:39.9038751Z * [new branch] gh/jansel/575/head -> origin/gh/jansel/575/head 2025-12-04T09:23:39.9039754Z * [new branch] gh/jansel/575/orig -> origin/gh/jansel/575/orig 2025-12-04T09:23:39.9040913Z * [new branch] gh/jansel/576/base -> origin/gh/jansel/576/base 2025-12-04T09:23:39.9041083Z * [new branch] gh/jansel/576/head -> origin/gh/jansel/576/head 2025-12-04T09:23:39.9044143Z * [new branch] gh/jansel/576/orig -> origin/gh/jansel/576/orig 2025-12-04T09:23:39.9044524Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-12-04T09:23:39.9044721Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-12-04T09:23:39.9044888Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-12-04T09:23:39.9045571Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-12-04T09:23:39.9046741Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-12-04T09:23:39.9046972Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-12-04T09:23:39.9053092Z * [new branch] gh/jerryzh168/1/base -> origin/gh/jerryzh168/1/base 2025-12-04T09:23:39.9053293Z * [new branch] gh/jerryzh168/1/head -> origin/gh/jerryzh168/1/head 2025-12-04T09:23:39.9053471Z * [new branch] gh/jerryzh168/1/orig -> origin/gh/jerryzh168/1/orig 2025-12-04T09:23:39.9053637Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-12-04T09:23:39.9053795Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-12-04T09:23:39.9053949Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-12-04T09:23:39.9054283Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-12-04T09:23:39.9054443Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-12-04T09:23:39.9054749Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-12-04T09:23:39.9056168Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-12-04T09:23:39.9056545Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-12-04T09:23:39.9056905Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-12-04T09:23:39.9059735Z * [new branch] gh/jiayisunx/77/base -> origin/gh/jiayisunx/77/base 2025-12-04T09:23:39.9060081Z * [new branch] gh/jiayisunx/77/head -> origin/gh/jiayisunx/77/head 2025-12-04T09:23:39.9060319Z * [new branch] gh/jiayisunx/77/orig -> origin/gh/jiayisunx/77/orig 2025-12-04T09:23:39.9060500Z * [new branch] gh/jiayisunx/78/base -> origin/gh/jiayisunx/78/base 2025-12-04T09:23:39.9060666Z * [new branch] gh/jiayisunx/78/head -> origin/gh/jiayisunx/78/head 2025-12-04T09:23:39.9062119Z * [new branch] gh/jiayisunx/78/orig -> origin/gh/jiayisunx/78/orig 2025-12-04T09:23:39.9062359Z * [new branch] gh/jiayisunx/79/base -> origin/gh/jiayisunx/79/base 2025-12-04T09:23:39.9064726Z * [new branch] gh/jiayisunx/79/head -> origin/gh/jiayisunx/79/head 2025-12-04T09:23:39.9065109Z * [new branch] gh/jiayisunx/79/orig -> origin/gh/jiayisunx/79/orig 2025-12-04T09:23:39.9065382Z * [new branch] gh/jiayisunx/82/base -> origin/gh/jiayisunx/82/base 2025-12-04T09:23:39.9065599Z * [new branch] gh/jiayisunx/82/head -> origin/gh/jiayisunx/82/head 2025-12-04T09:23:39.9066251Z * [new branch] gh/jiayisunx/82/orig -> origin/gh/jiayisunx/82/orig 2025-12-04T09:23:39.9070324Z * [new branch] gh/jiayisunx/83/base -> origin/gh/jiayisunx/83/base 2025-12-04T09:23:39.9070699Z * [new branch] gh/jiayisunx/83/head -> origin/gh/jiayisunx/83/head 2025-12-04T09:23:39.9070924Z * [new branch] gh/jiayisunx/83/orig -> origin/gh/jiayisunx/83/orig 2025-12-04T09:23:39.9071186Z * [new branch] gh/jiayisunx/84/base -> origin/gh/jiayisunx/84/base 2025-12-04T09:23:39.9071407Z * [new branch] gh/jiayisunx/84/head -> origin/gh/jiayisunx/84/head 2025-12-04T09:23:39.9073960Z * [new branch] gh/jiayisunx/84/orig -> origin/gh/jiayisunx/84/orig 2025-12-04T09:23:39.9074140Z * [new branch] gh/jiayisunx/85/base -> origin/gh/jiayisunx/85/base 2025-12-04T09:23:39.9074521Z * [new branch] gh/jiayisunx/85/head -> origin/gh/jiayisunx/85/head 2025-12-04T09:23:39.9075119Z * [new branch] gh/jiayisunx/85/orig -> origin/gh/jiayisunx/85/orig 2025-12-04T09:23:39.9078346Z * [new branch] gh/jiayisunx/86/base -> origin/gh/jiayisunx/86/base 2025-12-04T09:23:39.9078634Z * [new branch] gh/jiayisunx/86/head -> origin/gh/jiayisunx/86/head 2025-12-04T09:23:39.9078951Z * [new branch] gh/jiayisunx/86/orig -> origin/gh/jiayisunx/86/orig 2025-12-04T09:23:39.9079116Z * [new branch] gh/jiayisunx/87/base -> origin/gh/jiayisunx/87/base 2025-12-04T09:23:39.9079275Z * [new branch] gh/jiayisunx/87/head -> origin/gh/jiayisunx/87/head 2025-12-04T09:23:39.9079437Z * [new branch] gh/jiayisunx/87/orig -> origin/gh/jiayisunx/87/orig 2025-12-04T09:23:39.9079589Z * [new branch] gh/jiayisunx/88/base -> origin/gh/jiayisunx/88/base 2025-12-04T09:23:39.9079744Z * [new branch] gh/jiayisunx/88/head -> origin/gh/jiayisunx/88/head 2025-12-04T09:23:39.9079899Z * [new branch] gh/jiayisunx/88/orig -> origin/gh/jiayisunx/88/orig 2025-12-04T09:23:39.9081051Z * [new branch] gh/jiayisunx/89/base -> origin/gh/jiayisunx/89/base 2025-12-04T09:23:39.9086864Z * [new branch] gh/jiayisunx/89/head -> origin/gh/jiayisunx/89/head 2025-12-04T09:23:39.9087490Z * [new branch] gh/jiayisunx/89/orig -> origin/gh/jiayisunx/89/orig 2025-12-04T09:23:39.9087679Z * [new branch] gh/jiayisunx/90/base -> origin/gh/jiayisunx/90/base 2025-12-04T09:23:39.9087832Z * [new branch] gh/jiayisunx/90/head -> origin/gh/jiayisunx/90/head 2025-12-04T09:23:39.9087990Z * [new branch] gh/jiayisunx/90/orig -> origin/gh/jiayisunx/90/orig 2025-12-04T09:23:39.9088164Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-12-04T09:23:39.9088333Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-12-04T09:23:39.9090889Z * [new branch] gh/jturney/1/base -> origin/gh/jturney/1/base 2025-12-04T09:23:39.9091172Z * [new branch] gh/jturney/1/head -> origin/gh/jturney/1/head 2025-12-04T09:23:39.9091340Z * [new branch] gh/jturney/1/orig -> origin/gh/jturney/1/orig 2025-12-04T09:23:39.9091485Z * [new branch] gh/jturney/2/base -> origin/gh/jturney/2/base 2025-12-04T09:23:39.9091635Z * [new branch] gh/jturney/2/head -> origin/gh/jturney/2/head 2025-12-04T09:23:39.9091778Z * [new branch] gh/jturney/2/orig -> origin/gh/jturney/2/orig 2025-12-04T09:23:39.9096213Z * [new branch] gh/karthickai/10/base -> origin/gh/karthickai/10/base 2025-12-04T09:23:39.9096387Z * [new branch] gh/karthickai/10/head -> origin/gh/karthickai/10/head 2025-12-04T09:23:39.9096555Z * [new branch] gh/karthickai/10/orig -> origin/gh/karthickai/10/orig 2025-12-04T09:23:39.9096707Z * [new branch] gh/karthickai/11/base -> origin/gh/karthickai/11/base 2025-12-04T09:23:39.9096861Z * [new branch] gh/karthickai/11/head -> origin/gh/karthickai/11/head 2025-12-04T09:23:39.9097026Z * [new branch] gh/karthickai/11/orig -> origin/gh/karthickai/11/orig 2025-12-04T09:23:39.9097173Z * [new branch] gh/karthickai/12/base -> origin/gh/karthickai/12/base 2025-12-04T09:23:39.9097337Z * [new branch] gh/karthickai/12/head -> origin/gh/karthickai/12/head 2025-12-04T09:23:39.9102123Z * [new branch] gh/karthickai/12/orig -> origin/gh/karthickai/12/orig 2025-12-04T09:23:39.9102420Z * [new branch] gh/karthickai/13/base -> origin/gh/karthickai/13/base 2025-12-04T09:23:39.9102618Z * [new branch] gh/karthickai/13/head -> origin/gh/karthickai/13/head 2025-12-04T09:23:39.9102947Z * [new branch] gh/karthickai/13/orig -> origin/gh/karthickai/13/orig 2025-12-04T09:23:39.9103103Z * [new branch] gh/karthickai/14/base -> origin/gh/karthickai/14/base 2025-12-04T09:23:39.9103265Z * [new branch] gh/karthickai/14/head -> origin/gh/karthickai/14/head 2025-12-04T09:23:39.9103509Z * [new branch] gh/karthickai/14/orig -> origin/gh/karthickai/14/orig 2025-12-04T09:23:39.9106156Z * [new branch] gh/karthickai/15/base -> origin/gh/karthickai/15/base 2025-12-04T09:23:39.9106317Z * [new branch] gh/karthickai/15/head -> origin/gh/karthickai/15/head 2025-12-04T09:23:39.9106475Z * [new branch] gh/karthickai/15/orig -> origin/gh/karthickai/15/orig 2025-12-04T09:23:39.9106626Z * [new branch] gh/karthickai/16/base -> origin/gh/karthickai/16/base 2025-12-04T09:23:39.9108771Z * [new branch] gh/karthickai/16/head -> origin/gh/karthickai/16/head 2025-12-04T09:23:39.9110156Z * [new branch] gh/karthickai/16/orig -> origin/gh/karthickai/16/orig 2025-12-04T09:23:39.9110754Z * [new branch] gh/karthickai/17/base -> origin/gh/karthickai/17/base 2025-12-04T09:23:39.9110945Z * [new branch] gh/karthickai/17/head -> origin/gh/karthickai/17/head 2025-12-04T09:23:39.9111122Z * [new branch] gh/karthickai/17/orig -> origin/gh/karthickai/17/orig 2025-12-04T09:23:39.9111280Z * [new branch] gh/karthickai/18/base -> origin/gh/karthickai/18/base 2025-12-04T09:23:39.9111439Z * [new branch] gh/karthickai/18/head -> origin/gh/karthickai/18/head 2025-12-04T09:23:39.9111645Z * [new branch] gh/karthickai/18/orig -> origin/gh/karthickai/18/orig 2025-12-04T09:23:39.9113057Z * [new branch] gh/karthickai/19/base -> origin/gh/karthickai/19/base 2025-12-04T09:23:39.9113290Z * [new branch] gh/karthickai/19/head -> origin/gh/karthickai/19/head 2025-12-04T09:23:39.9114502Z * [new branch] gh/karthickai/19/orig -> origin/gh/karthickai/19/orig 2025-12-04T09:23:39.9117284Z * [new branch] gh/karthickai/20/base -> origin/gh/karthickai/20/base 2025-12-04T09:23:39.9118292Z * [new branch] gh/karthickai/20/head -> origin/gh/karthickai/20/head 2025-12-04T09:23:39.9118651Z * [new branch] gh/karthickai/20/orig -> origin/gh/karthickai/20/orig 2025-12-04T09:23:39.9120075Z * [new branch] gh/karthickai/21/base -> origin/gh/karthickai/21/base 2025-12-04T09:23:39.9120727Z * [new branch] gh/karthickai/21/head -> origin/gh/karthickai/21/head 2025-12-04T09:23:39.9122265Z * [new branch] gh/karthickai/21/orig -> origin/gh/karthickai/21/orig 2025-12-04T09:23:39.9122746Z * [new branch] gh/karthickai/22/base -> origin/gh/karthickai/22/base 2025-12-04T09:23:39.9125028Z * [new branch] gh/karthickai/22/head -> origin/gh/karthickai/22/head 2025-12-04T09:23:39.9125402Z * [new branch] gh/karthickai/22/orig -> origin/gh/karthickai/22/orig 2025-12-04T09:23:39.9125686Z * [new branch] gh/karthickai/23/base -> origin/gh/karthickai/23/base 2025-12-04T09:23:39.9126163Z * [new branch] gh/karthickai/23/head -> origin/gh/karthickai/23/head 2025-12-04T09:23:39.9130675Z * [new branch] gh/karthickai/23/orig -> origin/gh/karthickai/23/orig 2025-12-04T09:23:39.9130876Z * [new branch] gh/karthickai/24/base -> origin/gh/karthickai/24/base 2025-12-04T09:23:39.9131354Z * [new branch] gh/karthickai/24/head -> origin/gh/karthickai/24/head 2025-12-04T09:23:39.9131570Z * [new branch] gh/karthickai/24/orig -> origin/gh/karthickai/24/orig 2025-12-04T09:23:39.9131748Z * [new branch] gh/karthickai/25/base -> origin/gh/karthickai/25/base 2025-12-04T09:23:39.9132098Z * [new branch] gh/karthickai/25/head -> origin/gh/karthickai/25/head 2025-12-04T09:23:39.9132614Z * [new branch] gh/karthickai/25/orig -> origin/gh/karthickai/25/orig 2025-12-04T09:23:39.9134413Z * [new branch] gh/karthickai/26/base -> origin/gh/karthickai/26/base 2025-12-04T09:23:39.9134934Z * [new branch] gh/karthickai/26/head -> origin/gh/karthickai/26/head 2025-12-04T09:23:39.9135248Z * [new branch] gh/karthickai/26/orig -> origin/gh/karthickai/26/orig 2025-12-04T09:23:39.9140835Z * [new branch] gh/karthickai/6/base -> origin/gh/karthickai/6/base 2025-12-04T09:23:39.9141225Z * [new branch] gh/karthickai/6/head -> origin/gh/karthickai/6/head 2025-12-04T09:23:39.9141508Z * [new branch] gh/karthickai/6/orig -> origin/gh/karthickai/6/orig 2025-12-04T09:23:39.9141718Z * [new branch] gh/krocki/1/base -> origin/gh/krocki/1/base 2025-12-04T09:23:39.9141878Z * [new branch] gh/krocki/1/head -> origin/gh/krocki/1/head 2025-12-04T09:23:39.9146529Z * [new branch] gh/krocki/1/orig -> origin/gh/krocki/1/orig 2025-12-04T09:23:39.9146707Z * [new branch] gh/krocki/2/base -> origin/gh/krocki/2/base 2025-12-04T09:23:39.9146876Z * [new branch] gh/krocki/2/head -> origin/gh/krocki/2/head 2025-12-04T09:23:39.9151569Z * [new branch] gh/krocki/2/orig -> origin/gh/krocki/2/orig 2025-12-04T09:23:39.9151939Z * [new branch] gh/kurtamohler/60/base -> origin/gh/kurtamohler/60/base 2025-12-04T09:23:39.9152202Z * [new branch] gh/kurtamohler/60/head -> origin/gh/kurtamohler/60/head 2025-12-04T09:23:39.9152438Z * [new branch] gh/kurtamohler/60/orig -> origin/gh/kurtamohler/60/orig 2025-12-04T09:23:39.9152701Z * [new branch] gh/kurtamohler/61/base -> origin/gh/kurtamohler/61/base 2025-12-04T09:23:39.9157900Z * [new branch] gh/kurtamohler/61/head -> origin/gh/kurtamohler/61/head 2025-12-04T09:23:39.9158254Z * [new branch] gh/kurtamohler/61/orig -> origin/gh/kurtamohler/61/orig 2025-12-04T09:23:39.9158456Z * [new branch] gh/kurtamohler/62/base -> origin/gh/kurtamohler/62/base 2025-12-04T09:23:39.9158743Z * [new branch] gh/kurtamohler/62/head -> origin/gh/kurtamohler/62/head 2025-12-04T09:23:39.9158931Z * [new branch] gh/kurtamohler/62/orig -> origin/gh/kurtamohler/62/orig 2025-12-04T09:23:39.9159095Z * [new branch] gh/kurtamohler/63/base -> origin/gh/kurtamohler/63/base 2025-12-04T09:23:39.9159258Z * [new branch] gh/kurtamohler/63/head -> origin/gh/kurtamohler/63/head 2025-12-04T09:23:39.9159418Z * [new branch] gh/kurtamohler/63/orig -> origin/gh/kurtamohler/63/orig 2025-12-04T09:23:39.9159584Z * [new branch] gh/kurtamohler/64/base -> origin/gh/kurtamohler/64/base 2025-12-04T09:23:39.9159756Z * [new branch] gh/kurtamohler/64/head -> origin/gh/kurtamohler/64/head 2025-12-04T09:23:39.9160042Z * [new branch] gh/kurtamohler/64/orig -> origin/gh/kurtamohler/64/orig 2025-12-04T09:23:39.9161448Z * [new branch] gh/kurtamohler/65/base -> origin/gh/kurtamohler/65/base 2025-12-04T09:23:39.9162037Z * [new branch] gh/kurtamohler/65/head -> origin/gh/kurtamohler/65/head 2025-12-04T09:23:39.9162675Z * [new branch] gh/kurtamohler/65/orig -> origin/gh/kurtamohler/65/orig 2025-12-04T09:23:39.9165756Z * [new branch] gh/kurtamohler/66/base -> origin/gh/kurtamohler/66/base 2025-12-04T09:23:39.9165961Z * [new branch] gh/kurtamohler/66/head -> origin/gh/kurtamohler/66/head 2025-12-04T09:23:39.9166342Z * [new branch] gh/kurtamohler/66/orig -> origin/gh/kurtamohler/66/orig 2025-12-04T09:23:39.9166504Z * [new branch] gh/kurtamohler/67/base -> origin/gh/kurtamohler/67/base 2025-12-04T09:23:39.9169242Z * [new branch] gh/kurtamohler/67/head -> origin/gh/kurtamohler/67/head 2025-12-04T09:23:39.9169495Z * [new branch] gh/kurtamohler/67/orig -> origin/gh/kurtamohler/67/orig 2025-12-04T09:23:39.9169661Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-12-04T09:23:39.9173045Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-12-04T09:23:39.9173495Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-12-04T09:23:39.9173659Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-12-04T09:23:39.9173805Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-12-04T09:23:39.9173962Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-12-04T09:23:39.9178227Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-12-04T09:23:39.9178374Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-12-04T09:23:39.9178526Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-12-04T09:23:39.9178673Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-12-04T09:23:39.9178809Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-12-04T09:23:39.9178955Z * [new branch] gh/kwen2501/211/base -> origin/gh/kwen2501/211/base 2025-12-04T09:23:39.9181423Z * [new branch] gh/kwen2501/211/head -> origin/gh/kwen2501/211/head 2025-12-04T09:23:39.9181574Z * [new branch] gh/kwen2501/224/base -> origin/gh/kwen2501/224/base 2025-12-04T09:23:39.9181716Z * [new branch] gh/kwen2501/224/head -> origin/gh/kwen2501/224/head 2025-12-04T09:23:39.9181851Z * [new branch] gh/kwen2501/224/orig -> origin/gh/kwen2501/224/orig 2025-12-04T09:23:39.9183510Z * [new branch] gh/kwen2501/228/base -> origin/gh/kwen2501/228/base 2025-12-04T09:23:39.9183731Z * [new branch] gh/kwen2501/228/head -> origin/gh/kwen2501/228/head 2025-12-04T09:23:39.9184362Z * [new branch] gh/kwen2501/228/orig -> origin/gh/kwen2501/228/orig 2025-12-04T09:23:39.9188546Z * [new branch] gh/kwen2501/234/base -> origin/gh/kwen2501/234/base 2025-12-04T09:23:39.9188741Z * [new branch] gh/kwen2501/234/head -> origin/gh/kwen2501/234/head 2025-12-04T09:23:39.9188895Z * [new branch] gh/kwen2501/234/orig -> origin/gh/kwen2501/234/orig 2025-12-04T09:23:39.9189073Z * [new branch] gh/kwen2501/235/base -> origin/gh/kwen2501/235/base 2025-12-04T09:23:39.9189663Z * [new branch] gh/kwen2501/235/head -> origin/gh/kwen2501/235/head 2025-12-04T09:23:39.9189826Z * [new branch] gh/kwen2501/235/orig -> origin/gh/kwen2501/235/orig 2025-12-04T09:23:39.9190484Z * [new branch] gh/kwen2501/236/base -> origin/gh/kwen2501/236/base 2025-12-04T09:23:39.9195711Z * [new branch] gh/kwen2501/236/head -> origin/gh/kwen2501/236/head 2025-12-04T09:23:39.9195894Z * [new branch] gh/kwen2501/236/orig -> origin/gh/kwen2501/236/orig 2025-12-04T09:23:39.9196039Z * [new branch] gh/kwen2501/237/base -> origin/gh/kwen2501/237/base 2025-12-04T09:23:39.9196191Z * [new branch] gh/kwen2501/237/head -> origin/gh/kwen2501/237/head 2025-12-04T09:23:39.9196338Z * [new branch] gh/kwen2501/237/orig -> origin/gh/kwen2501/237/orig 2025-12-04T09:23:39.9196672Z * [new branch] gh/kwen2501/238/base -> origin/gh/kwen2501/238/base 2025-12-04T09:23:39.9196823Z * [new branch] gh/kwen2501/238/head -> origin/gh/kwen2501/238/head 2025-12-04T09:23:39.9198649Z * [new branch] gh/kwen2501/238/orig -> origin/gh/kwen2501/238/orig 2025-12-04T09:23:39.9199193Z * [new branch] gh/kwen2501/240/base -> origin/gh/kwen2501/240/base 2025-12-04T09:23:39.9199376Z * [new branch] gh/kwen2501/240/head -> origin/gh/kwen2501/240/head 2025-12-04T09:23:39.9199519Z * [new branch] gh/kwen2501/240/orig -> origin/gh/kwen2501/240/orig 2025-12-04T09:23:39.9199673Z * [new branch] gh/kwen2501/241/base -> origin/gh/kwen2501/241/base 2025-12-04T09:23:39.9200055Z * [new branch] gh/kwen2501/241/head -> origin/gh/kwen2501/241/head 2025-12-04T09:23:39.9200772Z * [new branch] gh/kwen2501/241/orig -> origin/gh/kwen2501/241/orig 2025-12-04T09:23:39.9207599Z * [new branch] gh/kwen2501/247/base -> origin/gh/kwen2501/247/base 2025-12-04T09:23:39.9207777Z * [new branch] gh/kwen2501/247/head -> origin/gh/kwen2501/247/head 2025-12-04T09:23:39.9207933Z * [new branch] gh/kwen2501/247/orig -> origin/gh/kwen2501/247/orig 2025-12-04T09:23:39.9208114Z * [new branch] gh/kwen2501/252/base -> origin/gh/kwen2501/252/base 2025-12-04T09:23:39.9208260Z * [new branch] gh/kwen2501/252/head -> origin/gh/kwen2501/252/head 2025-12-04T09:23:39.9208414Z * [new branch] gh/kwen2501/252/orig -> origin/gh/kwen2501/252/orig 2025-12-04T09:23:39.9210093Z * [new branch] gh/kwen2501/259/base -> origin/gh/kwen2501/259/base 2025-12-04T09:23:39.9210358Z * [new branch] gh/kwen2501/259/head -> origin/gh/kwen2501/259/head 2025-12-04T09:23:39.9210952Z * [new branch] gh/kwen2501/259/orig -> origin/gh/kwen2501/259/orig 2025-12-04T09:23:39.9211164Z * [new branch] gh/kwen2501/260/base -> origin/gh/kwen2501/260/base 2025-12-04T09:23:39.9211327Z * [new branch] gh/kwen2501/260/head -> origin/gh/kwen2501/260/head 2025-12-04T09:23:39.9211478Z * [new branch] gh/kwen2501/260/orig -> origin/gh/kwen2501/260/orig 2025-12-04T09:23:39.9211647Z * [new branch] gh/kwen2501/268/base -> origin/gh/kwen2501/268/base 2025-12-04T09:23:39.9212122Z * [new branch] gh/kwen2501/268/head -> origin/gh/kwen2501/268/head 2025-12-04T09:23:39.9212290Z * [new branch] gh/kwen2501/268/orig -> origin/gh/kwen2501/268/orig 2025-12-04T09:23:39.9213809Z * [new branch] gh/kwen2501/269/base -> origin/gh/kwen2501/269/base 2025-12-04T09:23:39.9214104Z * [new branch] gh/kwen2501/269/head -> origin/gh/kwen2501/269/head 2025-12-04T09:23:39.9214908Z * [new branch] gh/kwen2501/269/orig -> origin/gh/kwen2501/269/orig 2025-12-04T09:23:39.9216157Z * [new branch] gh/kwen2501/270/base -> origin/gh/kwen2501/270/base 2025-12-04T09:23:39.9216514Z * [new branch] gh/kwen2501/270/head -> origin/gh/kwen2501/270/head 2025-12-04T09:23:39.9217496Z * [new branch] gh/kwen2501/270/orig -> origin/gh/kwen2501/270/orig 2025-12-04T09:23:39.9218705Z * [new branch] gh/kwen2501/271/base -> origin/gh/kwen2501/271/base 2025-12-04T09:23:39.9219050Z * [new branch] gh/kwen2501/271/head -> origin/gh/kwen2501/271/head 2025-12-04T09:23:39.9257942Z * [new branch] gh/kwen2501/271/orig -> origin/gh/kwen2501/271/orig 2025-12-04T09:23:39.9258303Z * [new branch] gh/kwen2501/274/base -> origin/gh/kwen2501/274/base 2025-12-04T09:23:39.9258467Z * [new branch] gh/kwen2501/274/head -> origin/gh/kwen2501/274/head 2025-12-04T09:23:39.9258844Z * [new branch] gh/kwen2501/274/orig -> origin/gh/kwen2501/274/orig 2025-12-04T09:23:39.9258987Z * [new branch] gh/kwen2501/275/base -> origin/gh/kwen2501/275/base 2025-12-04T09:23:39.9259134Z * [new branch] gh/kwen2501/275/head -> origin/gh/kwen2501/275/head 2025-12-04T09:23:39.9259353Z * [new branch] gh/kwen2501/275/orig -> origin/gh/kwen2501/275/orig 2025-12-04T09:23:39.9259497Z * [new branch] gh/kwen2501/276/base -> origin/gh/kwen2501/276/base 2025-12-04T09:23:39.9259646Z * [new branch] gh/kwen2501/276/head -> origin/gh/kwen2501/276/head 2025-12-04T09:23:39.9259787Z * [new branch] gh/kwen2501/276/orig -> origin/gh/kwen2501/276/orig 2025-12-04T09:23:39.9259925Z * [new branch] gh/kwen2501/277/base -> origin/gh/kwen2501/277/base 2025-12-04T09:23:39.9260072Z * [new branch] gh/kwen2501/277/head -> origin/gh/kwen2501/277/head 2025-12-04T09:23:39.9260218Z * [new branch] gh/kwen2501/277/orig -> origin/gh/kwen2501/277/orig 2025-12-04T09:23:39.9260360Z * [new branch] gh/kwen2501/278/base -> origin/gh/kwen2501/278/base 2025-12-04T09:23:39.9260504Z * [new branch] gh/kwen2501/278/head -> origin/gh/kwen2501/278/head 2025-12-04T09:23:39.9260659Z * [new branch] gh/kwen2501/278/orig -> origin/gh/kwen2501/278/orig 2025-12-04T09:23:39.9260805Z * [new branch] gh/kwen2501/279/base -> origin/gh/kwen2501/279/base 2025-12-04T09:23:39.9260940Z * [new branch] gh/kwen2501/279/head -> origin/gh/kwen2501/279/head 2025-12-04T09:23:39.9261082Z * [new branch] gh/kwen2501/279/orig -> origin/gh/kwen2501/279/orig 2025-12-04T09:23:39.9261220Z * [new branch] gh/kwen2501/280/base -> origin/gh/kwen2501/280/base 2025-12-04T09:23:39.9261358Z * [new branch] gh/kwen2501/280/head -> origin/gh/kwen2501/280/head 2025-12-04T09:23:39.9261503Z * [new branch] gh/kwen2501/280/orig -> origin/gh/kwen2501/280/orig 2025-12-04T09:23:39.9261638Z * [new branch] gh/kwen2501/281/base -> origin/gh/kwen2501/281/base 2025-12-04T09:23:39.9261780Z * [new branch] gh/kwen2501/281/head -> origin/gh/kwen2501/281/head 2025-12-04T09:23:39.9261920Z * [new branch] gh/kwen2501/281/orig -> origin/gh/kwen2501/281/orig 2025-12-04T09:23:39.9262056Z * [new branch] gh/kwen2501/282/base -> origin/gh/kwen2501/282/base 2025-12-04T09:23:39.9262196Z * [new branch] gh/kwen2501/282/head -> origin/gh/kwen2501/282/head 2025-12-04T09:23:39.9262328Z * [new branch] gh/kwen2501/282/orig -> origin/gh/kwen2501/282/orig 2025-12-04T09:23:39.9262462Z * [new branch] gh/kwen2501/283/base -> origin/gh/kwen2501/283/base 2025-12-04T09:23:39.9262604Z * [new branch] gh/kwen2501/283/head -> origin/gh/kwen2501/283/head 2025-12-04T09:23:39.9262742Z * [new branch] gh/kwen2501/283/orig -> origin/gh/kwen2501/283/orig 2025-12-04T09:23:39.9262896Z * [new branch] gh/kwen2501/284/base -> origin/gh/kwen2501/284/base 2025-12-04T09:23:39.9263032Z * [new branch] gh/kwen2501/284/head -> origin/gh/kwen2501/284/head 2025-12-04T09:23:39.9263176Z * [new branch] gh/kwen2501/284/orig -> origin/gh/kwen2501/284/orig 2025-12-04T09:23:39.9263317Z * [new branch] gh/kwen2501/285/base -> origin/gh/kwen2501/285/base 2025-12-04T09:23:39.9263450Z * [new branch] gh/kwen2501/285/head -> origin/gh/kwen2501/285/head 2025-12-04T09:23:39.9263595Z * [new branch] gh/kwen2501/285/orig -> origin/gh/kwen2501/285/orig 2025-12-04T09:23:39.9263727Z * [new branch] gh/kwen2501/286/base -> origin/gh/kwen2501/286/base 2025-12-04T09:23:39.9263903Z * [new branch] gh/kwen2501/286/head -> origin/gh/kwen2501/286/head 2025-12-04T09:23:39.9264045Z * [new branch] gh/kwen2501/286/orig -> origin/gh/kwen2501/286/orig 2025-12-04T09:23:39.9264177Z * [new branch] gh/kwen2501/287/base -> origin/gh/kwen2501/287/base 2025-12-04T09:23:39.9264354Z * [new branch] gh/kwen2501/287/head -> origin/gh/kwen2501/287/head 2025-12-04T09:23:39.9264488Z * [new branch] gh/kwen2501/287/orig -> origin/gh/kwen2501/287/orig 2025-12-04T09:23:39.9264621Z * [new branch] gh/kwen2501/288/base -> origin/gh/kwen2501/288/base 2025-12-04T09:23:39.9264762Z * [new branch] gh/kwen2501/288/head -> origin/gh/kwen2501/288/head 2025-12-04T09:23:39.9264896Z * [new branch] gh/kwen2501/288/orig -> origin/gh/kwen2501/288/orig 2025-12-04T09:23:39.9265060Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-12-04T09:23:39.9265214Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-12-04T09:23:39.9265364Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-12-04T09:23:39.9265519Z * [new branch] gh/laithsakka/276/base -> origin/gh/laithsakka/276/base 2025-12-04T09:23:39.9265676Z * [new branch] gh/laithsakka/276/head -> origin/gh/laithsakka/276/head 2025-12-04T09:23:39.9265836Z * [new branch] gh/laithsakka/276/orig -> origin/gh/laithsakka/276/orig 2025-12-04T09:23:39.9265985Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-12-04T09:23:39.9266132Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-12-04T09:23:39.9266288Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-12-04T09:23:39.9266430Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-12-04T09:23:39.9268239Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-12-04T09:23:39.9268403Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-12-04T09:23:39.9275710Z * [new branch] gh/laithsakka/313/base -> origin/gh/laithsakka/313/base 2025-12-04T09:23:39.9277242Z * [new branch] gh/laithsakka/313/head -> origin/gh/laithsakka/313/head 2025-12-04T09:23:39.9277428Z * [new branch] gh/laithsakka/313/orig -> origin/gh/laithsakka/313/orig 2025-12-04T09:23:39.9278021Z * [new branch] gh/laithsakka/316/base -> origin/gh/laithsakka/316/base 2025-12-04T09:23:39.9278200Z * [new branch] gh/laithsakka/316/head -> origin/gh/laithsakka/316/head 2025-12-04T09:23:39.9278359Z * [new branch] gh/laithsakka/316/orig -> origin/gh/laithsakka/316/orig 2025-12-04T09:23:39.9278551Z * [new branch] gh/laithsakka/317/base -> origin/gh/laithsakka/317/base 2025-12-04T09:23:39.9278704Z * [new branch] gh/laithsakka/317/head -> origin/gh/laithsakka/317/head 2025-12-04T09:23:39.9278854Z * [new branch] gh/laithsakka/317/orig -> origin/gh/laithsakka/317/orig 2025-12-04T09:23:39.9279011Z * [new branch] gh/laithsakka/319/base -> origin/gh/laithsakka/319/base 2025-12-04T09:23:39.9279170Z * [new branch] gh/laithsakka/319/head -> origin/gh/laithsakka/319/head 2025-12-04T09:23:39.9279318Z * [new branch] gh/laithsakka/319/orig -> origin/gh/laithsakka/319/orig 2025-12-04T09:23:39.9279478Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-12-04T09:23:39.9279633Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-12-04T09:23:39.9280969Z * [new branch] gh/laithsakka/320/base -> origin/gh/laithsakka/320/base 2025-12-04T09:23:39.9285605Z * [new branch] gh/laithsakka/320/head -> origin/gh/laithsakka/320/head 2025-12-04T09:23:39.9285757Z * [new branch] gh/laithsakka/320/orig -> origin/gh/laithsakka/320/orig 2025-12-04T09:23:39.9285911Z * [new branch] gh/laithsakka/321/base -> origin/gh/laithsakka/321/base 2025-12-04T09:23:39.9286108Z * [new branch] gh/laithsakka/321/head -> origin/gh/laithsakka/321/head 2025-12-04T09:23:39.9286260Z * [new branch] gh/laithsakka/321/orig -> origin/gh/laithsakka/321/orig 2025-12-04T09:23:39.9286415Z * [new branch] gh/laithsakka/322/base -> origin/gh/laithsakka/322/base 2025-12-04T09:23:39.9286564Z * [new branch] gh/laithsakka/322/head -> origin/gh/laithsakka/322/head 2025-12-04T09:23:39.9286795Z * [new branch] gh/laithsakka/322/orig -> origin/gh/laithsakka/322/orig 2025-12-04T09:23:39.9290474Z * [new branch] gh/laithsakka/323/base -> origin/gh/laithsakka/323/base 2025-12-04T09:23:39.9293783Z * [new branch] gh/laithsakka/323/head -> origin/gh/laithsakka/323/head 2025-12-04T09:23:39.9294387Z * [new branch] gh/laithsakka/323/orig -> origin/gh/laithsakka/323/orig 2025-12-04T09:23:39.9294586Z * [new branch] gh/laithsakka/324/base -> origin/gh/laithsakka/324/base 2025-12-04T09:23:39.9294787Z * [new branch] gh/laithsakka/324/head -> origin/gh/laithsakka/324/head 2025-12-04T09:23:39.9294948Z * [new branch] gh/laithsakka/324/orig -> origin/gh/laithsakka/324/orig 2025-12-04T09:23:39.9295107Z * [new branch] gh/laithsakka/325/base -> origin/gh/laithsakka/325/base 2025-12-04T09:23:39.9295277Z * [new branch] gh/laithsakka/325/head -> origin/gh/laithsakka/325/head 2025-12-04T09:23:39.9295433Z * [new branch] gh/laithsakka/325/orig -> origin/gh/laithsakka/325/orig 2025-12-04T09:23:39.9295620Z * [new branch] gh/laithsakka/326/base -> origin/gh/laithsakka/326/base 2025-12-04T09:23:39.9298128Z * [new branch] gh/laithsakka/326/head -> origin/gh/laithsakka/326/head 2025-12-04T09:23:39.9298303Z * [new branch] gh/laithsakka/326/orig -> origin/gh/laithsakka/326/orig 2025-12-04T09:23:39.9298608Z * [new branch] gh/laithsakka/327/base -> origin/gh/laithsakka/327/base 2025-12-04T09:23:39.9298778Z * [new branch] gh/laithsakka/327/head -> origin/gh/laithsakka/327/head 2025-12-04T09:23:39.9298933Z * [new branch] gh/laithsakka/327/orig -> origin/gh/laithsakka/327/orig 2025-12-04T09:23:39.9301933Z * [new branch] gh/laithsakka/328/base -> origin/gh/laithsakka/328/base 2025-12-04T09:23:39.9302226Z * [new branch] gh/laithsakka/328/head -> origin/gh/laithsakka/328/head 2025-12-04T09:23:39.9302862Z * [new branch] gh/laithsakka/328/orig -> origin/gh/laithsakka/328/orig 2025-12-04T09:23:39.9303096Z * [new branch] gh/liangel/4/base -> origin/gh/liangel/4/base 2025-12-04T09:23:39.9307457Z * [new branch] gh/liangel/4/head -> origin/gh/liangel/4/head 2025-12-04T09:23:39.9307591Z * [new branch] gh/liangel/4/orig -> origin/gh/liangel/4/orig 2025-12-04T09:23:39.9307766Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-12-04T09:23:39.9307909Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-12-04T09:23:39.9311232Z * [new branch] gh/lw/4/base -> origin/gh/lw/4/base 2025-12-04T09:23:39.9311487Z * [new branch] gh/lw/4/head -> origin/gh/lw/4/head 2025-12-04T09:23:39.9311617Z * [new branch] gh/lw/4/orig -> origin/gh/lw/4/orig 2025-12-04T09:23:39.9311750Z * [new branch] gh/lw/5/base -> origin/gh/lw/5/base 2025-12-04T09:23:39.9312000Z * [new branch] gh/lw/5/head -> origin/gh/lw/5/head 2025-12-04T09:23:39.9316147Z * [new branch] gh/lw/5/orig -> origin/gh/lw/5/orig 2025-12-04T09:23:39.9316478Z * [new branch] gh/lw/6/base -> origin/gh/lw/6/base 2025-12-04T09:23:39.9316978Z * [new branch] gh/lw/6/head -> origin/gh/lw/6/head 2025-12-04T09:23:39.9317118Z * [new branch] gh/lw/6/orig -> origin/gh/lw/6/orig 2025-12-04T09:23:39.9317273Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-12-04T09:23:39.9317432Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-12-04T09:23:39.9317600Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-12-04T09:23:39.9319829Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-12-04T09:23:39.9320044Z * [new branch] gh/malfet/506/base -> origin/gh/malfet/506/base 2025-12-04T09:23:39.9320198Z * [new branch] gh/malfet/506/head -> origin/gh/malfet/506/head 2025-12-04T09:23:39.9320946Z * [new branch] gh/malfet/506/orig -> origin/gh/malfet/506/orig 2025-12-04T09:23:39.9322058Z * [new branch] gh/malfet/517/base -> origin/gh/malfet/517/base 2025-12-04T09:23:39.9322514Z * [new branch] gh/malfet/517/head -> origin/gh/malfet/517/head 2025-12-04T09:23:39.9323671Z * [new branch] gh/malfet/528/base -> origin/gh/malfet/528/base 2025-12-04T09:23:39.9323934Z * [new branch] gh/malfet/528/head -> origin/gh/malfet/528/head 2025-12-04T09:23:39.9325075Z * [new branch] gh/malfet/528/orig -> origin/gh/malfet/528/orig 2025-12-04T09:23:39.9326743Z * [new branch] gh/malfet/537/base -> origin/gh/malfet/537/base 2025-12-04T09:23:39.9327449Z * [new branch] gh/malfet/537/head -> origin/gh/malfet/537/head 2025-12-04T09:23:39.9327629Z * [new branch] gh/malfet/537/orig -> origin/gh/malfet/537/orig 2025-12-04T09:23:39.9328212Z * [new branch] gh/malfet/546/base -> origin/gh/malfet/546/base 2025-12-04T09:23:39.9328858Z * [new branch] gh/malfet/546/head -> origin/gh/malfet/546/head 2025-12-04T09:23:39.9329830Z * [new branch] gh/malfet/546/orig -> origin/gh/malfet/546/orig 2025-12-04T09:23:39.9330453Z * [new branch] gh/malfet/565/base -> origin/gh/malfet/565/base 2025-12-04T09:23:39.9331478Z * [new branch] gh/malfet/565/head -> origin/gh/malfet/565/head 2025-12-04T09:23:39.9332566Z * [new branch] gh/malfet/565/orig -> origin/gh/malfet/565/orig 2025-12-04T09:23:39.9333876Z * [new branch] gh/malfet/575/base -> origin/gh/malfet/575/base 2025-12-04T09:23:39.9334140Z * [new branch] gh/malfet/575/head -> origin/gh/malfet/575/head 2025-12-04T09:23:39.9335748Z * [new branch] gh/malfet/575/orig -> origin/gh/malfet/575/orig 2025-12-04T09:23:39.9336018Z * [new branch] gh/malfet/580/base -> origin/gh/malfet/580/base 2025-12-04T09:23:39.9336361Z * [new branch] gh/malfet/580/head -> origin/gh/malfet/580/head 2025-12-04T09:23:39.9337354Z * [new branch] gh/malfet/580/orig -> origin/gh/malfet/580/orig 2025-12-04T09:23:39.9338365Z * [new branch] gh/malfet/581/base -> origin/gh/malfet/581/base 2025-12-04T09:23:39.9338992Z * [new branch] gh/malfet/581/head -> origin/gh/malfet/581/head 2025-12-04T09:23:39.9339596Z * [new branch] gh/malfet/581/orig -> origin/gh/malfet/581/orig 2025-12-04T09:23:39.9340759Z * [new branch] gh/malfet/583/base -> origin/gh/malfet/583/base 2025-12-04T09:23:39.9341311Z * [new branch] gh/malfet/583/head -> origin/gh/malfet/583/head 2025-12-04T09:23:39.9342207Z * [new branch] gh/malfet/583/orig -> origin/gh/malfet/583/orig 2025-12-04T09:23:39.9347447Z * [new branch] gh/malfet/586/base -> origin/gh/malfet/586/base 2025-12-04T09:23:39.9347761Z * [new branch] gh/malfet/586/head -> origin/gh/malfet/586/head 2025-12-04T09:23:39.9347916Z * [new branch] gh/malfet/586/orig -> origin/gh/malfet/586/orig 2025-12-04T09:23:39.9348065Z * [new branch] gh/malfet/587/base -> origin/gh/malfet/587/base 2025-12-04T09:23:39.9348209Z * [new branch] gh/malfet/587/head -> origin/gh/malfet/587/head 2025-12-04T09:23:39.9348356Z * [new branch] gh/malfet/587/orig -> origin/gh/malfet/587/orig 2025-12-04T09:23:39.9348511Z * [new branch] gh/malfet/588/base -> origin/gh/malfet/588/base 2025-12-04T09:23:39.9348663Z * [new branch] gh/malfet/588/head -> origin/gh/malfet/588/head 2025-12-04T09:23:39.9348843Z * [new branch] gh/malfet/588/orig -> origin/gh/malfet/588/orig 2025-12-04T09:23:39.9352333Z * [new branch] gh/malfet/589/base -> origin/gh/malfet/589/base 2025-12-04T09:23:39.9352535Z * [new branch] gh/malfet/589/head -> origin/gh/malfet/589/head 2025-12-04T09:23:39.9352677Z * [new branch] gh/malfet/589/orig -> origin/gh/malfet/589/orig 2025-12-04T09:23:39.9352839Z * [new branch] gh/malfet/590/base -> origin/gh/malfet/590/base 2025-12-04T09:23:39.9353016Z * [new branch] gh/malfet/590/head -> origin/gh/malfet/590/head 2025-12-04T09:23:39.9354894Z * [new branch] gh/malfet/590/orig -> origin/gh/malfet/590/orig 2025-12-04T09:23:39.9355234Z * [new branch] gh/malfet/591/base -> origin/gh/malfet/591/base 2025-12-04T09:23:39.9355617Z * [new branch] gh/malfet/591/head -> origin/gh/malfet/591/head 2025-12-04T09:23:39.9357947Z * [new branch] gh/malfet/591/orig -> origin/gh/malfet/591/orig 2025-12-04T09:23:39.9358129Z * [new branch] gh/malfet/592/base -> origin/gh/malfet/592/base 2025-12-04T09:23:39.9358296Z * [new branch] gh/malfet/592/head -> origin/gh/malfet/592/head 2025-12-04T09:23:39.9358801Z * [new branch] gh/malfet/592/orig -> origin/gh/malfet/592/orig 2025-12-04T09:23:39.9360041Z * [new branch] gh/malfet/593/base -> origin/gh/malfet/593/base 2025-12-04T09:23:39.9360565Z * [new branch] gh/malfet/593/head -> origin/gh/malfet/593/head 2025-12-04T09:23:39.9361863Z * [new branch] gh/malfet/593/orig -> origin/gh/malfet/593/orig 2025-12-04T09:23:39.9362390Z * [new branch] gh/malfet/594/base -> origin/gh/malfet/594/base 2025-12-04T09:23:39.9363676Z * [new branch] gh/malfet/594/head -> origin/gh/malfet/594/head 2025-12-04T09:23:39.9364289Z * [new branch] gh/malfet/594/orig -> origin/gh/malfet/594/orig 2025-12-04T09:23:39.9364666Z * [new branch] gh/malfet/595/base -> origin/gh/malfet/595/base 2025-12-04T09:23:39.9367347Z * [new branch] gh/malfet/595/head -> origin/gh/malfet/595/head 2025-12-04T09:23:39.9367527Z * [new branch] gh/malfet/595/orig -> origin/gh/malfet/595/orig 2025-12-04T09:23:39.9367686Z * [new branch] gh/malfet/596/base -> origin/gh/malfet/596/base 2025-12-04T09:23:39.9368191Z * [new branch] gh/malfet/596/head -> origin/gh/malfet/596/head 2025-12-04T09:23:39.9368351Z * [new branch] gh/malfet/596/orig -> origin/gh/malfet/596/orig 2025-12-04T09:23:39.9370521Z * [new branch] gh/malfet/597/base -> origin/gh/malfet/597/base 2025-12-04T09:23:39.9370681Z * [new branch] gh/malfet/597/head -> origin/gh/malfet/597/head 2025-12-04T09:23:39.9373008Z * [new branch] gh/malfet/597/orig -> origin/gh/malfet/597/orig 2025-12-04T09:23:39.9378489Z * [new branch] gh/malfet/598/base -> origin/gh/malfet/598/base 2025-12-04T09:23:39.9378850Z * [new branch] gh/malfet/598/head -> origin/gh/malfet/598/head 2025-12-04T09:23:39.9379011Z * [new branch] gh/malfet/598/orig -> origin/gh/malfet/598/orig 2025-12-04T09:23:39.9379157Z * [new branch] gh/malfet/599/base -> origin/gh/malfet/599/base 2025-12-04T09:23:39.9379312Z * [new branch] gh/malfet/599/head -> origin/gh/malfet/599/head 2025-12-04T09:23:39.9379451Z * [new branch] gh/malfet/599/orig -> origin/gh/malfet/599/orig 2025-12-04T09:23:39.9379609Z * [new branch] gh/malfet/600/base -> origin/gh/malfet/600/base 2025-12-04T09:23:39.9379746Z * [new branch] gh/malfet/600/head -> origin/gh/malfet/600/head 2025-12-04T09:23:39.9379884Z * [new branch] gh/malfet/600/orig -> origin/gh/malfet/600/orig 2025-12-04T09:23:39.9380041Z * [new branch] gh/malfet/601/base -> origin/gh/malfet/601/base 2025-12-04T09:23:39.9380183Z * [new branch] gh/malfet/601/head -> origin/gh/malfet/601/head 2025-12-04T09:23:39.9380738Z * [new branch] gh/malfet/601/orig -> origin/gh/malfet/601/orig 2025-12-04T09:23:39.9383419Z * [new branch] gh/malfet/602/base -> origin/gh/malfet/602/base 2025-12-04T09:23:39.9383747Z * [new branch] gh/malfet/602/head -> origin/gh/malfet/602/head 2025-12-04T09:23:39.9383924Z * [new branch] gh/malfet/602/orig -> origin/gh/malfet/602/orig 2025-12-04T09:23:39.9388531Z * [new branch] gh/malfet/603/base -> origin/gh/malfet/603/base 2025-12-04T09:23:39.9388720Z * [new branch] gh/malfet/603/head -> origin/gh/malfet/603/head 2025-12-04T09:23:39.9388876Z * [new branch] gh/malfet/603/orig -> origin/gh/malfet/603/orig 2025-12-04T09:23:39.9389034Z * [new branch] gh/malfet/604/base -> origin/gh/malfet/604/base 2025-12-04T09:23:39.9389179Z * [new branch] gh/malfet/604/head -> origin/gh/malfet/604/head 2025-12-04T09:23:39.9394312Z * [new branch] gh/malfet/604/orig -> origin/gh/malfet/604/orig 2025-12-04T09:23:39.9394658Z * [new branch] gh/malfet/605/base -> origin/gh/malfet/605/base 2025-12-04T09:23:39.9394858Z * [new branch] gh/malfet/605/head -> origin/gh/malfet/605/head 2025-12-04T09:23:39.9395150Z * [new branch] gh/malfet/605/orig -> origin/gh/malfet/605/orig 2025-12-04T09:23:39.9395346Z * [new branch] gh/malfet/606/base -> origin/gh/malfet/606/base 2025-12-04T09:23:39.9395627Z * [new branch] gh/malfet/606/head -> origin/gh/malfet/606/head 2025-12-04T09:23:39.9396347Z * [new branch] gh/malfet/606/orig -> origin/gh/malfet/606/orig 2025-12-04T09:23:39.9396547Z * [new branch] gh/malfet/607/base -> origin/gh/malfet/607/base 2025-12-04T09:23:39.9398113Z * [new branch] gh/malfet/607/head -> origin/gh/malfet/607/head 2025-12-04T09:23:39.9398367Z * [new branch] gh/malfet/607/orig -> origin/gh/malfet/607/orig 2025-12-04T09:23:39.9398531Z * [new branch] gh/malfet/608/base -> origin/gh/malfet/608/base 2025-12-04T09:23:39.9398681Z * [new branch] gh/malfet/608/head -> origin/gh/malfet/608/head 2025-12-04T09:23:39.9398832Z * [new branch] gh/malfet/608/orig -> origin/gh/malfet/608/orig 2025-12-04T09:23:39.9399132Z * [new branch] gh/malfet/609/base -> origin/gh/malfet/609/base 2025-12-04T09:23:39.9399281Z * [new branch] gh/malfet/609/head -> origin/gh/malfet/609/head 2025-12-04T09:23:39.9399456Z * [new branch] gh/malfet/609/orig -> origin/gh/malfet/609/orig 2025-12-04T09:23:39.9400098Z * [new branch] gh/malfet/610/base -> origin/gh/malfet/610/base 2025-12-04T09:23:39.9401737Z * [new branch] gh/malfet/610/head -> origin/gh/malfet/610/head 2025-12-04T09:23:39.9401899Z * [new branch] gh/malfet/610/orig -> origin/gh/malfet/610/orig 2025-12-04T09:23:39.9403180Z * [new branch] gh/malfet/611/base -> origin/gh/malfet/611/base 2025-12-04T09:23:39.9403353Z * [new branch] gh/malfet/611/head -> origin/gh/malfet/611/head 2025-12-04T09:23:39.9403990Z * [new branch] gh/malfet/611/orig -> origin/gh/malfet/611/orig 2025-12-04T09:23:39.9408684Z * [new branch] gh/malfet/612/base -> origin/gh/malfet/612/base 2025-12-04T09:23:39.9408865Z * [new branch] gh/malfet/612/head -> origin/gh/malfet/612/head 2025-12-04T09:23:39.9409017Z * [new branch] gh/malfet/612/orig -> origin/gh/malfet/612/orig 2025-12-04T09:23:39.9409177Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-12-04T09:23:39.9409317Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-12-04T09:23:39.9409507Z * [new branch] gh/manuelcandales/11/base -> origin/gh/manuelcandales/11/base 2025-12-04T09:23:39.9409773Z * [new branch] gh/manuelcandales/11/head -> origin/gh/manuelcandales/11/head 2025-12-04T09:23:39.9411023Z * [new branch] gh/manuelcandales/11/orig -> origin/gh/manuelcandales/11/orig 2025-12-04T09:23:39.9412323Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-12-04T09:23:39.9413800Z * [new branch] gh/masnesral/1/base -> origin/gh/masnesral/1/base 2025-12-04T09:23:39.9414181Z * [new branch] gh/masnesral/1/head -> origin/gh/masnesral/1/head 2025-12-04T09:23:39.9415186Z * [new branch] gh/masnesral/1/orig -> origin/gh/masnesral/1/orig 2025-12-04T09:23:39.9416449Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-12-04T09:23:39.9416921Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-12-04T09:23:39.9418346Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-12-04T09:23:39.9418806Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-12-04T09:23:39.9420827Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-12-04T09:23:39.9421135Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-12-04T09:23:39.9421289Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-12-04T09:23:39.9425255Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-12-04T09:23:39.9425436Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-12-04T09:23:39.9425603Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-12-04T09:23:39.9425821Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-12-04T09:23:39.9425984Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-12-04T09:23:39.9426160Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-12-04T09:23:39.9426309Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-12-04T09:23:39.9429375Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-12-04T09:23:39.9429580Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-12-04T09:23:39.9429761Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-12-04T09:23:39.9430137Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-12-04T09:23:39.9433499Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-12-04T09:23:39.9433681Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-12-04T09:23:39.9433868Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-12-04T09:23:39.9434047Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-12-04T09:23:39.9434247Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-12-04T09:23:39.9435267Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-12-04T09:23:39.9436239Z * [new branch] gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base 2025-12-04T09:23:39.9436663Z * [new branch] gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head 2025-12-04T09:23:39.9437693Z * [new branch] gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig 2025-12-04T09:23:39.9440824Z * [new branch] gh/mikaylagawarecki/341/base -> origin/gh/mikaylagawarecki/341/base 2025-12-04T09:23:39.9441036Z * [new branch] gh/mikaylagawarecki/341/head -> origin/gh/mikaylagawarecki/341/head 2025-12-04T09:23:39.9441237Z * [new branch] gh/mikaylagawarecki/341/orig -> origin/gh/mikaylagawarecki/341/orig 2025-12-04T09:23:39.9441422Z * [new branch] gh/mikaylagawarecki/342/base -> origin/gh/mikaylagawarecki/342/base 2025-12-04T09:23:39.9445429Z * [new branch] gh/mikaylagawarecki/342/head -> origin/gh/mikaylagawarecki/342/head 2025-12-04T09:23:39.9445623Z * [new branch] gh/mikaylagawarecki/342/orig -> origin/gh/mikaylagawarecki/342/orig 2025-12-04T09:23:39.9445817Z * [new branch] gh/mikaylagawarecki/345/base -> origin/gh/mikaylagawarecki/345/base 2025-12-04T09:23:39.9446065Z * [new branch] gh/mikaylagawarecki/345/head -> origin/gh/mikaylagawarecki/345/head 2025-12-04T09:23:39.9446251Z * [new branch] gh/mikaylagawarecki/345/orig -> origin/gh/mikaylagawarecki/345/orig 2025-12-04T09:23:39.9451886Z * [new branch] gh/mikaylagawarecki/346/base -> origin/gh/mikaylagawarecki/346/base 2025-12-04T09:23:39.9456762Z * [new branch] gh/mikaylagawarecki/346/head -> origin/gh/mikaylagawarecki/346/head 2025-12-04T09:23:39.9462280Z * [new branch] gh/mikaylagawarecki/346/orig -> origin/gh/mikaylagawarecki/346/orig 2025-12-04T09:23:39.9467208Z * [new branch] gh/mikaylagawarecki/347/base -> origin/gh/mikaylagawarecki/347/base 2025-12-04T09:23:39.9472284Z * [new branch] gh/mikaylagawarecki/347/head -> origin/gh/mikaylagawarecki/347/head 2025-12-04T09:23:39.9472512Z * [new branch] gh/mikaylagawarecki/347/orig -> origin/gh/mikaylagawarecki/347/orig 2025-12-04T09:23:39.9472711Z * [new branch] gh/mikaylagawarecki/350/base -> origin/gh/mikaylagawarecki/350/base 2025-12-04T09:23:39.9472922Z * [new branch] gh/mikaylagawarecki/350/head -> origin/gh/mikaylagawarecki/350/head 2025-12-04T09:23:39.9473096Z * [new branch] gh/mikaylagawarecki/350/orig -> origin/gh/mikaylagawarecki/350/orig 2025-12-04T09:23:39.9473269Z * [new branch] gh/mikaylagawarecki/351/base -> origin/gh/mikaylagawarecki/351/base 2025-12-04T09:23:39.9473451Z * [new branch] gh/mikaylagawarecki/351/head -> origin/gh/mikaylagawarecki/351/head 2025-12-04T09:23:39.9473880Z * [new branch] gh/mikaylagawarecki/351/orig -> origin/gh/mikaylagawarecki/351/orig 2025-12-04T09:23:39.9474058Z * [new branch] gh/mikaylagawarecki/352/base -> origin/gh/mikaylagawarecki/352/base 2025-12-04T09:23:39.9474242Z * [new branch] gh/mikaylagawarecki/352/head -> origin/gh/mikaylagawarecki/352/head 2025-12-04T09:23:39.9474506Z * [new branch] gh/mikaylagawarecki/352/orig -> origin/gh/mikaylagawarecki/352/orig 2025-12-04T09:23:39.9474694Z * [new branch] gh/mikaylagawarecki/353/base -> origin/gh/mikaylagawarecki/353/base 2025-12-04T09:23:39.9474871Z * [new branch] gh/mikaylagawarecki/353/head -> origin/gh/mikaylagawarecki/353/head 2025-12-04T09:23:39.9475046Z * [new branch] gh/mikaylagawarecki/353/orig -> origin/gh/mikaylagawarecki/353/orig 2025-12-04T09:23:39.9475230Z * [new branch] gh/mikaylagawarecki/354/base -> origin/gh/mikaylagawarecki/354/base 2025-12-04T09:23:39.9475408Z * [new branch] gh/mikaylagawarecki/354/head -> origin/gh/mikaylagawarecki/354/head 2025-12-04T09:23:39.9475588Z * [new branch] gh/mikaylagawarecki/354/orig -> origin/gh/mikaylagawarecki/354/orig 2025-12-04T09:23:39.9475763Z * [new branch] gh/mikaylagawarecki/356/base -> origin/gh/mikaylagawarecki/356/base 2025-12-04T09:23:39.9475939Z * [new branch] gh/mikaylagawarecki/356/head -> origin/gh/mikaylagawarecki/356/head 2025-12-04T09:23:39.9476125Z * [new branch] gh/mikaylagawarecki/356/orig -> origin/gh/mikaylagawarecki/356/orig 2025-12-04T09:23:39.9476300Z * [new branch] gh/mikaylagawarecki/357/base -> origin/gh/mikaylagawarecki/357/base 2025-12-04T09:23:39.9476483Z * [new branch] gh/mikaylagawarecki/357/head -> origin/gh/mikaylagawarecki/357/head 2025-12-04T09:23:39.9476659Z * [new branch] gh/mikaylagawarecki/357/orig -> origin/gh/mikaylagawarecki/357/orig 2025-12-04T09:23:39.9476838Z * [new branch] gh/mikaylagawarecki/359/base -> origin/gh/mikaylagawarecki/359/base 2025-12-04T09:23:39.9477021Z * [new branch] gh/mikaylagawarecki/359/head -> origin/gh/mikaylagawarecki/359/head 2025-12-04T09:23:39.9477196Z * [new branch] gh/mikaylagawarecki/359/orig -> origin/gh/mikaylagawarecki/359/orig 2025-12-04T09:23:39.9477382Z * [new branch] gh/mikaylagawarecki/360/base -> origin/gh/mikaylagawarecki/360/base 2025-12-04T09:23:39.9477556Z * [new branch] gh/mikaylagawarecki/360/head -> origin/gh/mikaylagawarecki/360/head 2025-12-04T09:23:39.9477738Z * [new branch] gh/mikaylagawarecki/360/orig -> origin/gh/mikaylagawarecki/360/orig 2025-12-04T09:23:39.9477919Z * [new branch] gh/mikaylagawarecki/361/base -> origin/gh/mikaylagawarecki/361/base 2025-12-04T09:23:39.9478094Z * [new branch] gh/mikaylagawarecki/361/head -> origin/gh/mikaylagawarecki/361/head 2025-12-04T09:23:39.9478281Z * [new branch] gh/mikaylagawarecki/361/orig -> origin/gh/mikaylagawarecki/361/orig 2025-12-04T09:23:39.9478456Z * [new branch] gh/mikaylagawarecki/362/base -> origin/gh/mikaylagawarecki/362/base 2025-12-04T09:23:39.9478631Z * [new branch] gh/mikaylagawarecki/362/head -> origin/gh/mikaylagawarecki/362/head 2025-12-04T09:23:39.9478818Z * [new branch] gh/mikaylagawarecki/362/orig -> origin/gh/mikaylagawarecki/362/orig 2025-12-04T09:23:39.9480190Z * [new branch] gh/mikaylagawarecki/363/base -> origin/gh/mikaylagawarecki/363/base 2025-12-04T09:23:39.9480989Z * [new branch] gh/mikaylagawarecki/363/head -> origin/gh/mikaylagawarecki/363/head 2025-12-04T09:23:39.9481549Z * [new branch] gh/mikaylagawarecki/363/orig -> origin/gh/mikaylagawarecki/363/orig 2025-12-04T09:23:39.9484293Z * [new branch] gh/mikaylagawarecki/364/base -> origin/gh/mikaylagawarecki/364/base 2025-12-04T09:23:39.9484728Z * [new branch] gh/mikaylagawarecki/364/head -> origin/gh/mikaylagawarecki/364/head 2025-12-04T09:23:39.9484923Z * [new branch] gh/mikaylagawarecki/364/orig -> origin/gh/mikaylagawarecki/364/orig 2025-12-04T09:23:39.9486134Z * [new branch] gh/mikaylagawarecki/365/base -> origin/gh/mikaylagawarecki/365/base 2025-12-04T09:23:39.9492296Z * [new branch] gh/mikaylagawarecki/365/head -> origin/gh/mikaylagawarecki/365/head 2025-12-04T09:23:39.9492695Z * [new branch] gh/mikaylagawarecki/365/orig -> origin/gh/mikaylagawarecki/365/orig 2025-12-04T09:23:39.9492973Z * [new branch] gh/mikaylagawarecki/366/base -> origin/gh/mikaylagawarecki/366/base 2025-12-04T09:23:39.9493178Z * [new branch] gh/mikaylagawarecki/366/head -> origin/gh/mikaylagawarecki/366/head 2025-12-04T09:23:39.9493355Z * [new branch] gh/mikaylagawarecki/366/orig -> origin/gh/mikaylagawarecki/366/orig 2025-12-04T09:23:39.9493658Z * [new branch] gh/mikaylagawarecki/367/base -> origin/gh/mikaylagawarecki/367/base 2025-12-04T09:23:39.9496030Z * [new branch] gh/mikaylagawarecki/367/head -> origin/gh/mikaylagawarecki/367/head 2025-12-04T09:23:39.9496292Z * [new branch] gh/mikaylagawarecki/367/orig -> origin/gh/mikaylagawarecki/367/orig 2025-12-04T09:23:39.9496482Z * [new branch] gh/mikaylagawarecki/368/base -> origin/gh/mikaylagawarecki/368/base 2025-12-04T09:23:39.9496729Z * [new branch] gh/mikaylagawarecki/368/head -> origin/gh/mikaylagawarecki/368/head 2025-12-04T09:23:39.9496987Z * [new branch] gh/mikaylagawarecki/368/orig -> origin/gh/mikaylagawarecki/368/orig 2025-12-04T09:23:39.9497220Z * [new branch] gh/mikaylagawarecki/369/base -> origin/gh/mikaylagawarecki/369/base 2025-12-04T09:23:39.9497405Z * [new branch] gh/mikaylagawarecki/369/head -> origin/gh/mikaylagawarecki/369/head 2025-12-04T09:23:39.9497652Z * [new branch] gh/mikaylagawarecki/369/orig -> origin/gh/mikaylagawarecki/369/orig 2025-12-04T09:23:39.9501427Z * [new branch] gh/mikaylagawarecki/370/base -> origin/gh/mikaylagawarecki/370/base 2025-12-04T09:23:39.9501678Z * [new branch] gh/mikaylagawarecki/370/head -> origin/gh/mikaylagawarecki/370/head 2025-12-04T09:23:39.9501872Z * [new branch] gh/mikaylagawarecki/370/orig -> origin/gh/mikaylagawarecki/370/orig 2025-12-04T09:23:39.9502133Z * [new branch] gh/mikaylagawarecki/371/base -> origin/gh/mikaylagawarecki/371/base 2025-12-04T09:23:39.9502316Z * [new branch] gh/mikaylagawarecki/371/head -> origin/gh/mikaylagawarecki/371/head 2025-12-04T09:23:39.9505596Z * [new branch] gh/mikaylagawarecki/371/orig -> origin/gh/mikaylagawarecki/371/orig 2025-12-04T09:23:39.9506223Z * [new branch] gh/mikaylagawarecki/372/base -> origin/gh/mikaylagawarecki/372/base 2025-12-04T09:23:39.9506431Z * [new branch] gh/mikaylagawarecki/372/head -> origin/gh/mikaylagawarecki/372/head 2025-12-04T09:23:39.9506631Z * [new branch] gh/mikaylagawarecki/372/orig -> origin/gh/mikaylagawarecki/372/orig 2025-12-04T09:23:39.9506800Z * [new branch] gh/mikaylagawarecki/373/base -> origin/gh/mikaylagawarecki/373/base 2025-12-04T09:23:39.9506981Z * [new branch] gh/mikaylagawarecki/373/head -> origin/gh/mikaylagawarecki/373/head 2025-12-04T09:23:39.9507178Z * [new branch] gh/mikaylagawarecki/373/orig -> origin/gh/mikaylagawarecki/373/orig 2025-12-04T09:23:39.9507804Z * [new branch] gh/mikaylagawarecki/374/base -> origin/gh/mikaylagawarecki/374/base 2025-12-04T09:23:39.9508571Z * [new branch] gh/mikaylagawarecki/374/head -> origin/gh/mikaylagawarecki/374/head 2025-12-04T09:23:39.9509088Z * [new branch] gh/mikaylagawarecki/374/orig -> origin/gh/mikaylagawarecki/374/orig 2025-12-04T09:23:39.9510347Z * [new branch] gh/mikaylagawarecki/375/base -> origin/gh/mikaylagawarecki/375/base 2025-12-04T09:23:39.9510878Z * [new branch] gh/mikaylagawarecki/375/head -> origin/gh/mikaylagawarecki/375/head 2025-12-04T09:23:39.9511805Z * [new branch] gh/mikaylagawarecki/375/orig -> origin/gh/mikaylagawarecki/375/orig 2025-12-04T09:23:39.9512843Z * [new branch] gh/mikaylagawarecki/376/base -> origin/gh/mikaylagawarecki/376/base 2025-12-04T09:23:39.9513469Z * [new branch] gh/mikaylagawarecki/376/head -> origin/gh/mikaylagawarecki/376/head 2025-12-04T09:23:39.9514104Z * [new branch] gh/mikaylagawarecki/376/orig -> origin/gh/mikaylagawarecki/376/orig 2025-12-04T09:23:39.9515354Z * [new branch] gh/mikaylagawarecki/377/base -> origin/gh/mikaylagawarecki/377/base 2025-12-04T09:23:39.9515876Z * [new branch] gh/mikaylagawarecki/377/head -> origin/gh/mikaylagawarecki/377/head 2025-12-04T09:23:39.9516889Z * [new branch] gh/mikaylagawarecki/377/orig -> origin/gh/mikaylagawarecki/377/orig 2025-12-04T09:23:39.9517877Z * [new branch] gh/mikaylagawarecki/378/base -> origin/gh/mikaylagawarecki/378/base 2025-12-04T09:23:39.9518454Z * [new branch] gh/mikaylagawarecki/378/head -> origin/gh/mikaylagawarecki/378/head 2025-12-04T09:23:39.9519365Z * [new branch] gh/mikaylagawarecki/378/orig -> origin/gh/mikaylagawarecki/378/orig 2025-12-04T09:23:39.9520469Z * [new branch] gh/mikaylagawarecki/379/base -> origin/gh/mikaylagawarecki/379/base 2025-12-04T09:23:39.9520867Z * [new branch] gh/mikaylagawarecki/379/head -> origin/gh/mikaylagawarecki/379/head 2025-12-04T09:23:39.9522589Z * [new branch] gh/mikaylagawarecki/379/orig -> origin/gh/mikaylagawarecki/379/orig 2025-12-04T09:23:39.9522807Z * [new branch] gh/mikaylagawarecki/380/base -> origin/gh/mikaylagawarecki/380/base 2025-12-04T09:23:39.9523421Z * [new branch] gh/mikaylagawarecki/380/head -> origin/gh/mikaylagawarecki/380/head 2025-12-04T09:23:39.9524099Z * [new branch] gh/mikaylagawarecki/380/orig -> origin/gh/mikaylagawarecki/380/orig 2025-12-04T09:23:39.9525358Z * [new branch] gh/mikaylagawarecki/381/base -> origin/gh/mikaylagawarecki/381/base 2025-12-04T09:23:39.9525607Z * [new branch] gh/mikaylagawarecki/381/head -> origin/gh/mikaylagawarecki/381/head 2025-12-04T09:23:39.9526730Z * [new branch] gh/mikaylagawarecki/381/orig -> origin/gh/mikaylagawarecki/381/orig 2025-12-04T09:23:39.9527398Z * [new branch] gh/mikaylagawarecki/382/base -> origin/gh/mikaylagawarecki/382/base 2025-12-04T09:23:39.9528049Z * [new branch] gh/mikaylagawarecki/382/head -> origin/gh/mikaylagawarecki/382/head 2025-12-04T09:23:39.9528921Z * [new branch] gh/mikaylagawarecki/382/orig -> origin/gh/mikaylagawarecki/382/orig 2025-12-04T09:23:39.9530030Z * [new branch] gh/mikaylagawarecki/383/base -> origin/gh/mikaylagawarecki/383/base 2025-12-04T09:23:39.9530521Z * [new branch] gh/mikaylagawarecki/383/head -> origin/gh/mikaylagawarecki/383/head 2025-12-04T09:23:39.9531486Z * [new branch] gh/mikaylagawarecki/383/orig -> origin/gh/mikaylagawarecki/383/orig 2025-12-04T09:23:39.9536002Z * [new branch] gh/mikaylagawarecki/384/base -> origin/gh/mikaylagawarecki/384/base 2025-12-04T09:23:39.9536262Z * [new branch] gh/mikaylagawarecki/384/head -> origin/gh/mikaylagawarecki/384/head 2025-12-04T09:23:39.9536455Z * [new branch] gh/mikaylagawarecki/384/orig -> origin/gh/mikaylagawarecki/384/orig 2025-12-04T09:23:39.9536691Z * [new branch] gh/mikaylagawarecki/385/base -> origin/gh/mikaylagawarecki/385/base 2025-12-04T09:23:39.9536877Z * [new branch] gh/mikaylagawarecki/385/head -> origin/gh/mikaylagawarecki/385/head 2025-12-04T09:23:39.9537103Z * [new branch] gh/mikaylagawarecki/385/orig -> origin/gh/mikaylagawarecki/385/orig 2025-12-04T09:23:39.9537456Z * [new branch] gh/mikaylagawarecki/386/base -> origin/gh/mikaylagawarecki/386/base 2025-12-04T09:23:39.9538186Z * [new branch] gh/mikaylagawarecki/386/head -> origin/gh/mikaylagawarecki/386/head 2025-12-04T09:23:39.9538898Z * [new branch] gh/mikaylagawarecki/386/orig -> origin/gh/mikaylagawarecki/386/orig 2025-12-04T09:23:39.9540734Z * [new branch] gh/mikaylagawarecki/387/base -> origin/gh/mikaylagawarecki/387/base 2025-12-04T09:23:39.9540936Z * [new branch] gh/mikaylagawarecki/387/head -> origin/gh/mikaylagawarecki/387/head 2025-12-04T09:23:39.9541380Z * [new branch] gh/mikaylagawarecki/387/orig -> origin/gh/mikaylagawarecki/387/orig 2025-12-04T09:23:39.9542570Z * [new branch] gh/mikaylagawarecki/388/base -> origin/gh/mikaylagawarecki/388/base 2025-12-04T09:23:39.9542870Z * [new branch] gh/mikaylagawarecki/388/head -> origin/gh/mikaylagawarecki/388/head 2025-12-04T09:23:39.9543903Z * [new branch] gh/mikaylagawarecki/388/orig -> origin/gh/mikaylagawarecki/388/orig 2025-12-04T09:23:39.9545055Z * [new branch] gh/mikaylagawarecki/389/base -> origin/gh/mikaylagawarecki/389/base 2025-12-04T09:23:39.9545389Z * [new branch] gh/mikaylagawarecki/389/head -> origin/gh/mikaylagawarecki/389/head 2025-12-04T09:23:39.9546537Z * [new branch] gh/mikaylagawarecki/389/orig -> origin/gh/mikaylagawarecki/389/orig 2025-12-04T09:23:39.9547570Z * [new branch] gh/mikaylagawarecki/390/base -> origin/gh/mikaylagawarecki/390/base 2025-12-04T09:23:39.9553065Z * [new branch] gh/mikaylagawarecki/390/head -> origin/gh/mikaylagawarecki/390/head 2025-12-04T09:23:39.9557931Z * [new branch] gh/mikaylagawarecki/390/orig -> origin/gh/mikaylagawarecki/390/orig 2025-12-04T09:23:39.9558164Z * [new branch] gh/mikaylagawarecki/391/base -> origin/gh/mikaylagawarecki/391/base 2025-12-04T09:23:39.9558351Z * [new branch] gh/mikaylagawarecki/391/head -> origin/gh/mikaylagawarecki/391/head 2025-12-04T09:23:39.9558577Z * [new branch] gh/mikaylagawarecki/391/orig -> origin/gh/mikaylagawarecki/391/orig 2025-12-04T09:23:39.9558763Z * [new branch] gh/mikaylagawarecki/392/base -> origin/gh/mikaylagawarecki/392/base 2025-12-04T09:23:39.9558966Z * [new branch] gh/mikaylagawarecki/392/head -> origin/gh/mikaylagawarecki/392/head 2025-12-04T09:23:39.9559156Z * [new branch] gh/mikaylagawarecki/392/orig -> origin/gh/mikaylagawarecki/392/orig 2025-12-04T09:23:39.9559506Z * [new branch] gh/mlazos/41/base -> origin/gh/mlazos/41/base 2025-12-04T09:23:39.9560563Z * [new branch] gh/mlazos/41/head -> origin/gh/mlazos/41/head 2025-12-04T09:23:39.9561037Z * [new branch] gh/mlazos/41/orig -> origin/gh/mlazos/41/orig 2025-12-04T09:23:39.9565258Z * [new branch] gh/mlazos/42/base -> origin/gh/mlazos/42/base 2025-12-04T09:23:39.9565416Z * [new branch] gh/mlazos/42/head -> origin/gh/mlazos/42/head 2025-12-04T09:23:39.9565596Z * [new branch] gh/mlazos/42/orig -> origin/gh/mlazos/42/orig 2025-12-04T09:23:39.9565744Z * [new branch] gh/mlazos/43/base -> origin/gh/mlazos/43/base 2025-12-04T09:23:39.9565899Z * [new branch] gh/mlazos/43/head -> origin/gh/mlazos/43/head 2025-12-04T09:23:39.9569415Z * [new branch] gh/mlazos/43/orig -> origin/gh/mlazos/43/orig 2025-12-04T09:23:39.9569572Z * [new branch] gh/mlazos/44/base -> origin/gh/mlazos/44/base 2025-12-04T09:23:39.9569720Z * [new branch] gh/mlazos/44/head -> origin/gh/mlazos/44/head 2025-12-04T09:23:39.9569865Z * [new branch] gh/mlazos/44/orig -> origin/gh/mlazos/44/orig 2025-12-04T09:23:39.9570027Z * [new branch] gh/mlazos/47/base -> origin/gh/mlazos/47/base 2025-12-04T09:23:39.9570363Z * [new branch] gh/mlazos/47/head -> origin/gh/mlazos/47/head 2025-12-04T09:23:39.9573822Z * [new branch] gh/mlazos/47/orig -> origin/gh/mlazos/47/orig 2025-12-04T09:23:39.9574160Z * [new branch] gh/mlazos/48/base -> origin/gh/mlazos/48/base 2025-12-04T09:23:39.9574636Z * [new branch] gh/mlazos/48/head -> origin/gh/mlazos/48/head 2025-12-04T09:23:39.9575084Z * [new branch] gh/mlazos/48/orig -> origin/gh/mlazos/48/orig 2025-12-04T09:23:39.9575233Z * [new branch] gh/mlazos/49/base -> origin/gh/mlazos/49/base 2025-12-04T09:23:39.9575360Z * [new branch] gh/mlazos/49/head -> origin/gh/mlazos/49/head 2025-12-04T09:23:39.9580654Z * [new branch] gh/mlazos/49/orig -> origin/gh/mlazos/49/orig 2025-12-04T09:23:39.9580981Z * [new branch] gh/mlazos/50/base -> origin/gh/mlazos/50/base 2025-12-04T09:23:39.9581215Z * [new branch] gh/mlazos/50/head -> origin/gh/mlazos/50/head 2025-12-04T09:23:39.9581375Z * [new branch] gh/mlazos/50/orig -> origin/gh/mlazos/50/orig 2025-12-04T09:23:39.9581512Z * [new branch] gh/mlazos/51/base -> origin/gh/mlazos/51/base 2025-12-04T09:23:39.9581782Z * [new branch] gh/mlazos/51/head -> origin/gh/mlazos/51/head 2025-12-04T09:23:39.9583042Z * [new branch] gh/mlazos/51/orig -> origin/gh/mlazos/51/orig 2025-12-04T09:23:39.9583199Z * [new branch] gh/mlazos/52/base -> origin/gh/mlazos/52/base 2025-12-04T09:23:39.9583333Z * [new branch] gh/mlazos/52/head -> origin/gh/mlazos/52/head 2025-12-04T09:23:39.9583462Z * [new branch] gh/mlazos/52/orig -> origin/gh/mlazos/52/orig 2025-12-04T09:23:39.9583734Z * [new branch] gh/mlazos/53/base -> origin/gh/mlazos/53/base 2025-12-04T09:23:39.9583891Z * [new branch] gh/mlazos/53/head -> origin/gh/mlazos/53/head 2025-12-04T09:23:39.9584107Z * [new branch] gh/mlazos/53/orig -> origin/gh/mlazos/53/orig 2025-12-04T09:23:39.9584255Z * [new branch] gh/mlazos/54/base -> origin/gh/mlazos/54/base 2025-12-04T09:23:39.9589933Z * [new branch] gh/mlazos/54/head -> origin/gh/mlazos/54/head 2025-12-04T09:23:39.9590268Z * [new branch] gh/mlazos/54/orig -> origin/gh/mlazos/54/orig 2025-12-04T09:23:39.9590432Z * [new branch] gh/mlazos/55/base -> origin/gh/mlazos/55/base 2025-12-04T09:23:39.9590563Z * [new branch] gh/mlazos/55/head -> origin/gh/mlazos/55/head 2025-12-04T09:23:39.9590697Z * [new branch] gh/mlazos/55/orig -> origin/gh/mlazos/55/orig 2025-12-04T09:23:39.9590963Z * [new branch] gh/mlazos/56/base -> origin/gh/mlazos/56/base 2025-12-04T09:23:39.9591121Z * [new branch] gh/mlazos/56/head -> origin/gh/mlazos/56/head 2025-12-04T09:23:39.9592198Z * [new branch] gh/mlazos/56/orig -> origin/gh/mlazos/56/orig 2025-12-04T09:23:39.9592351Z * [new branch] gh/mlazos/57/base -> origin/gh/mlazos/57/base 2025-12-04T09:23:39.9592499Z * [new branch] gh/mlazos/57/head -> origin/gh/mlazos/57/head 2025-12-04T09:23:39.9592629Z * [new branch] gh/mlazos/57/orig -> origin/gh/mlazos/57/orig 2025-12-04T09:23:39.9592888Z * [new branch] gh/mlazos/58/base -> origin/gh/mlazos/58/base 2025-12-04T09:23:39.9593039Z * [new branch] gh/mlazos/58/head -> origin/gh/mlazos/58/head 2025-12-04T09:23:39.9593249Z * [new branch] gh/mlazos/58/orig -> origin/gh/mlazos/58/orig 2025-12-04T09:23:39.9593489Z * [new branch] gh/mlazos/59/base -> origin/gh/mlazos/59/base 2025-12-04T09:23:39.9597705Z * [new branch] gh/mlazos/59/head -> origin/gh/mlazos/59/head 2025-12-04T09:23:39.9597883Z * [new branch] gh/mlazos/59/orig -> origin/gh/mlazos/59/orig 2025-12-04T09:23:39.9598032Z * [new branch] gh/mlazos/60/base -> origin/gh/mlazos/60/base 2025-12-04T09:23:39.9598338Z * [new branch] gh/mlazos/60/head -> origin/gh/mlazos/60/head 2025-12-04T09:23:39.9598491Z * [new branch] gh/mlazos/60/orig -> origin/gh/mlazos/60/orig 2025-12-04T09:23:39.9598647Z * [new branch] gh/mlazos/61/base -> origin/gh/mlazos/61/base 2025-12-04T09:23:39.9598977Z * [new branch] gh/mlazos/61/head -> origin/gh/mlazos/61/head 2025-12-04T09:23:39.9599128Z * [new branch] gh/mlazos/61/orig -> origin/gh/mlazos/61/orig 2025-12-04T09:23:39.9601334Z * [new branch] gh/mlazos/62/base -> origin/gh/mlazos/62/base 2025-12-04T09:23:39.9609486Z * [new branch] gh/mlazos/62/head -> origin/gh/mlazos/62/head 2025-12-04T09:23:39.9609668Z * [new branch] gh/mlazos/62/orig -> origin/gh/mlazos/62/orig 2025-12-04T09:23:39.9609814Z * [new branch] gh/mlazos/63/base -> origin/gh/mlazos/63/base 2025-12-04T09:23:39.9609973Z * [new branch] gh/mlazos/63/head -> origin/gh/mlazos/63/head 2025-12-04T09:23:39.9610124Z * [new branch] gh/mlazos/63/orig -> origin/gh/mlazos/63/orig 2025-12-04T09:23:39.9610266Z * [new branch] gh/mlazos/64/base -> origin/gh/mlazos/64/base 2025-12-04T09:23:39.9610407Z * [new branch] gh/mlazos/64/head -> origin/gh/mlazos/64/head 2025-12-04T09:23:39.9610549Z * [new branch] gh/mlazos/64/orig -> origin/gh/mlazos/64/orig 2025-12-04T09:23:39.9611050Z * [new branch] gh/mlazos/65/base -> origin/gh/mlazos/65/base 2025-12-04T09:23:39.9611211Z * [new branch] gh/mlazos/65/head -> origin/gh/mlazos/65/head 2025-12-04T09:23:39.9611350Z * [new branch] gh/mlazos/65/orig -> origin/gh/mlazos/65/orig 2025-12-04T09:23:39.9611489Z * [new branch] gh/mlazos/66/base -> origin/gh/mlazos/66/base 2025-12-04T09:23:39.9611640Z * [new branch] gh/mlazos/66/head -> origin/gh/mlazos/66/head 2025-12-04T09:23:39.9611772Z * [new branch] gh/mlazos/66/orig -> origin/gh/mlazos/66/orig 2025-12-04T09:23:39.9611974Z * [new branch] gh/mlazos/67/base -> origin/gh/mlazos/67/base 2025-12-04T09:23:39.9612229Z * [new branch] gh/mlazos/67/head -> origin/gh/mlazos/67/head 2025-12-04T09:23:39.9614596Z * [new branch] gh/mlazos/67/orig -> origin/gh/mlazos/67/orig 2025-12-04T09:23:39.9614938Z * [new branch] gh/mlazos/68/base -> origin/gh/mlazos/68/base 2025-12-04T09:23:39.9615204Z * [new branch] gh/mlazos/68/head -> origin/gh/mlazos/68/head 2025-12-04T09:23:39.9615369Z * [new branch] gh/mlazos/68/orig -> origin/gh/mlazos/68/orig 2025-12-04T09:23:39.9619990Z * [new branch] gh/mlazos/69/base -> origin/gh/mlazos/69/base 2025-12-04T09:23:39.9620194Z * [new branch] gh/mlazos/69/head -> origin/gh/mlazos/69/head 2025-12-04T09:23:39.9620363Z * [new branch] gh/mlazos/69/orig -> origin/gh/mlazos/69/orig 2025-12-04T09:23:39.9620513Z * [new branch] gh/mlazos/70/base -> origin/gh/mlazos/70/base 2025-12-04T09:23:39.9620651Z * [new branch] gh/mlazos/70/head -> origin/gh/mlazos/70/head 2025-12-04T09:23:39.9620797Z * [new branch] gh/mlazos/70/orig -> origin/gh/mlazos/70/orig 2025-12-04T09:23:39.9621286Z * [new branch] gh/mlazos/71/base -> origin/gh/mlazos/71/base 2025-12-04T09:23:39.9622016Z * [new branch] gh/mlazos/71/head -> origin/gh/mlazos/71/head 2025-12-04T09:23:39.9622633Z * [new branch] gh/mlazos/71/orig -> origin/gh/mlazos/71/orig 2025-12-04T09:23:39.9626834Z * [new branch] gh/mlazos/72/base -> origin/gh/mlazos/72/base 2025-12-04T09:23:39.9627180Z * [new branch] gh/mlazos/72/head -> origin/gh/mlazos/72/head 2025-12-04T09:23:39.9627338Z * [new branch] gh/mlazos/72/orig -> origin/gh/mlazos/72/orig 2025-12-04T09:23:39.9627485Z * [new branch] gh/mlazos/73/base -> origin/gh/mlazos/73/base 2025-12-04T09:23:39.9627633Z * [new branch] gh/mlazos/73/head -> origin/gh/mlazos/73/head 2025-12-04T09:23:39.9630169Z * [new branch] gh/mlazos/73/orig -> origin/gh/mlazos/73/orig 2025-12-04T09:23:39.9630361Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-12-04T09:23:39.9630526Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-12-04T09:23:39.9631493Z * [new branch] gh/muchulee8/73/base -> origin/gh/muchulee8/73/base 2025-12-04T09:23:39.9632088Z * [new branch] gh/muchulee8/73/head -> origin/gh/muchulee8/73/head 2025-12-04T09:23:39.9633038Z * [new branch] gh/muchulee8/73/orig -> origin/gh/muchulee8/73/orig 2025-12-04T09:23:39.9634332Z * [new branch] gh/naveenthangudu/1/base -> origin/gh/naveenthangudu/1/base 2025-12-04T09:23:39.9634823Z * [new branch] gh/naveenthangudu/1/head -> origin/gh/naveenthangudu/1/head 2025-12-04T09:23:39.9635851Z * [new branch] gh/naveenthangudu/1/orig -> origin/gh/naveenthangudu/1/orig 2025-12-04T09:23:39.9637279Z * [new branch] gh/naveenthangudu/2/base -> origin/gh/naveenthangudu/2/base 2025-12-04T09:23:39.9637578Z * [new branch] gh/naveenthangudu/2/head -> origin/gh/naveenthangudu/2/head 2025-12-04T09:23:39.9637811Z * [new branch] gh/naveenthangudu/2/orig -> origin/gh/naveenthangudu/2/orig 2025-12-04T09:23:39.9640106Z * [new branch] gh/naveenthangudu/3/base -> origin/gh/naveenthangudu/3/base 2025-12-04T09:23:39.9640515Z * [new branch] gh/naveenthangudu/3/head -> origin/gh/naveenthangudu/3/head 2025-12-04T09:23:39.9640737Z * [new branch] gh/naveenthangudu/3/orig -> origin/gh/naveenthangudu/3/orig 2025-12-04T09:23:39.9644783Z * [new branch] gh/naveenthangudu/4/base -> origin/gh/naveenthangudu/4/base 2025-12-04T09:23:39.9645056Z * [new branch] gh/naveenthangudu/4/head -> origin/gh/naveenthangudu/4/head 2025-12-04T09:23:39.9645223Z * [new branch] gh/naveenthangudu/4/orig -> origin/gh/naveenthangudu/4/orig 2025-12-04T09:23:39.9645395Z * [new branch] gh/naveenthangudu/5/base -> origin/gh/naveenthangudu/5/base 2025-12-04T09:23:39.9645569Z * [new branch] gh/naveenthangudu/5/head -> origin/gh/naveenthangudu/5/head 2025-12-04T09:23:39.9651080Z * [new branch] gh/naveenthangudu/5/orig -> origin/gh/naveenthangudu/5/orig 2025-12-04T09:23:39.9657528Z * [new branch] gh/naveenthangudu/6/base -> origin/gh/naveenthangudu/6/base 2025-12-04T09:23:39.9659421Z * [new branch] gh/naveenthangudu/6/head -> origin/gh/naveenthangudu/6/head 2025-12-04T09:23:39.9659745Z * [new branch] gh/naveenthangudu/6/orig -> origin/gh/naveenthangudu/6/orig 2025-12-04T09:23:39.9666041Z * [new branch] gh/naveenthangudu/7/base -> origin/gh/naveenthangudu/7/base 2025-12-04T09:23:39.9666255Z * [new branch] gh/naveenthangudu/7/head -> origin/gh/naveenthangudu/7/head 2025-12-04T09:23:39.9666414Z * [new branch] gh/naveenthangudu/7/orig -> origin/gh/naveenthangudu/7/orig 2025-12-04T09:23:39.9666567Z * [new branch] gh/naveenthangudu/8/base -> origin/gh/naveenthangudu/8/base 2025-12-04T09:23:39.9666951Z * [new branch] gh/naveenthangudu/8/head -> origin/gh/naveenthangudu/8/head 2025-12-04T09:23:39.9667112Z * [new branch] gh/naveenthangudu/8/orig -> origin/gh/naveenthangudu/8/orig 2025-12-04T09:23:39.9667341Z * [new branch] gh/naveenthangudu/9/base -> origin/gh/naveenthangudu/9/base 2025-12-04T09:23:39.9667501Z * [new branch] gh/naveenthangudu/9/head -> origin/gh/naveenthangudu/9/head 2025-12-04T09:23:39.9667651Z * [new branch] gh/naveenthangudu/9/orig -> origin/gh/naveenthangudu/9/orig 2025-12-04T09:23:39.9667804Z * [new branch] gh/nikitaved/1/base -> origin/gh/nikitaved/1/base 2025-12-04T09:23:39.9667947Z * [new branch] gh/nikitaved/1/head -> origin/gh/nikitaved/1/head 2025-12-04T09:23:39.9668091Z * [new branch] gh/nikitaved/1/orig -> origin/gh/nikitaved/1/orig 2025-12-04T09:23:39.9668247Z * [new branch] gh/nikitaved/10/base -> origin/gh/nikitaved/10/base 2025-12-04T09:23:39.9668401Z * [new branch] gh/nikitaved/10/head -> origin/gh/nikitaved/10/head 2025-12-04T09:23:39.9668542Z * [new branch] gh/nikitaved/10/orig -> origin/gh/nikitaved/10/orig 2025-12-04T09:23:39.9668680Z * [new branch] gh/nikitaved/11/base -> origin/gh/nikitaved/11/base 2025-12-04T09:23:39.9668819Z * [new branch] gh/nikitaved/11/head -> origin/gh/nikitaved/11/head 2025-12-04T09:23:39.9668964Z * [new branch] gh/nikitaved/11/orig -> origin/gh/nikitaved/11/orig 2025-12-04T09:23:39.9669099Z * [new branch] gh/nikitaved/12/base -> origin/gh/nikitaved/12/base 2025-12-04T09:23:39.9669243Z * [new branch] gh/nikitaved/12/head -> origin/gh/nikitaved/12/head 2025-12-04T09:23:39.9669377Z * [new branch] gh/nikitaved/12/orig -> origin/gh/nikitaved/12/orig 2025-12-04T09:23:39.9669514Z * [new branch] gh/nikitaved/13/base -> origin/gh/nikitaved/13/base 2025-12-04T09:23:39.9669654Z * [new branch] gh/nikitaved/13/head -> origin/gh/nikitaved/13/head 2025-12-04T09:23:39.9669786Z * [new branch] gh/nikitaved/13/orig -> origin/gh/nikitaved/13/orig 2025-12-04T09:23:39.9669930Z * [new branch] gh/nikitaved/14/base -> origin/gh/nikitaved/14/base 2025-12-04T09:23:39.9670068Z * [new branch] gh/nikitaved/14/head -> origin/gh/nikitaved/14/head 2025-12-04T09:23:39.9670200Z * [new branch] gh/nikitaved/14/orig -> origin/gh/nikitaved/14/orig 2025-12-04T09:23:39.9675905Z * [new branch] gh/nikitaved/15/base -> origin/gh/nikitaved/15/base 2025-12-04T09:23:39.9677287Z * [new branch] gh/nikitaved/15/head -> origin/gh/nikitaved/15/head 2025-12-04T09:23:39.9677448Z * [new branch] gh/nikitaved/15/orig -> origin/gh/nikitaved/15/orig 2025-12-04T09:23:39.9677604Z * [new branch] gh/nikitaved/16/base -> origin/gh/nikitaved/16/base 2025-12-04T09:23:39.9677742Z * [new branch] gh/nikitaved/16/head -> origin/gh/nikitaved/16/head 2025-12-04T09:23:39.9677901Z * [new branch] gh/nikitaved/16/orig -> origin/gh/nikitaved/16/orig 2025-12-04T09:23:39.9678060Z * [new branch] gh/nikitaved/2/base -> origin/gh/nikitaved/2/base 2025-12-04T09:23:39.9678207Z * [new branch] gh/nikitaved/2/head -> origin/gh/nikitaved/2/head 2025-12-04T09:23:39.9678350Z * [new branch] gh/nikitaved/2/orig -> origin/gh/nikitaved/2/orig 2025-12-04T09:23:39.9678492Z * [new branch] gh/nikitaved/4/base -> origin/gh/nikitaved/4/base 2025-12-04T09:23:39.9678640Z * [new branch] gh/nikitaved/4/head -> origin/gh/nikitaved/4/head 2025-12-04T09:23:39.9678918Z * [new branch] gh/nikitaved/4/orig -> origin/gh/nikitaved/4/orig 2025-12-04T09:23:39.9679070Z * [new branch] gh/nikitaved/5/base -> origin/gh/nikitaved/5/base 2025-12-04T09:23:39.9679215Z * [new branch] gh/nikitaved/5/head -> origin/gh/nikitaved/5/head 2025-12-04T09:23:39.9679422Z * [new branch] gh/nikitaved/5/orig -> origin/gh/nikitaved/5/orig 2025-12-04T09:23:39.9679915Z * [new branch] gh/nikitaved/6/base -> origin/gh/nikitaved/6/base 2025-12-04T09:23:39.9681148Z * [new branch] gh/nikitaved/6/head -> origin/gh/nikitaved/6/head 2025-12-04T09:23:39.9687954Z * [new branch] gh/nikitaved/6/orig -> origin/gh/nikitaved/6/orig 2025-12-04T09:23:39.9688330Z * [new branch] gh/nikitaved/8/base -> origin/gh/nikitaved/8/base 2025-12-04T09:23:39.9688577Z * [new branch] gh/nikitaved/8/head -> origin/gh/nikitaved/8/head 2025-12-04T09:23:39.9688754Z * [new branch] gh/nikitaved/8/orig -> origin/gh/nikitaved/8/orig 2025-12-04T09:23:39.9688896Z * [new branch] gh/nikitaved/9/base -> origin/gh/nikitaved/9/base 2025-12-04T09:23:39.9689033Z * [new branch] gh/nikitaved/9/head -> origin/gh/nikitaved/9/head 2025-12-04T09:23:39.9689317Z * [new branch] gh/nikitaved/9/orig -> origin/gh/nikitaved/9/orig 2025-12-04T09:23:39.9689950Z * [new branch] gh/oulgen/10/base -> origin/gh/oulgen/10/base 2025-12-04T09:23:39.9690124Z * [new branch] gh/oulgen/10/head -> origin/gh/oulgen/10/head 2025-12-04T09:23:39.9690258Z * [new branch] gh/oulgen/10/orig -> origin/gh/oulgen/10/orig 2025-12-04T09:23:39.9690394Z * [new branch] gh/oulgen/11/base -> origin/gh/oulgen/11/base 2025-12-04T09:23:39.9690667Z * [new branch] gh/oulgen/11/head -> origin/gh/oulgen/11/head 2025-12-04T09:23:39.9691017Z * [new branch] gh/oulgen/11/orig -> origin/gh/oulgen/11/orig 2025-12-04T09:23:39.9692822Z * [new branch] gh/oulgen/12/base -> origin/gh/oulgen/12/base 2025-12-04T09:23:39.9693165Z * [new branch] gh/oulgen/12/head -> origin/gh/oulgen/12/head 2025-12-04T09:23:39.9693346Z * [new branch] gh/oulgen/12/orig -> origin/gh/oulgen/12/orig 2025-12-04T09:23:39.9694935Z * [new branch] gh/oulgen/13/base -> origin/gh/oulgen/13/base 2025-12-04T09:23:39.9695261Z * [new branch] gh/oulgen/13/head -> origin/gh/oulgen/13/head 2025-12-04T09:23:39.9695648Z * [new branch] gh/oulgen/13/orig -> origin/gh/oulgen/13/orig 2025-12-04T09:23:39.9698009Z * [new branch] gh/oulgen/14/base -> origin/gh/oulgen/14/base 2025-12-04T09:23:39.9698191Z * [new branch] gh/oulgen/14/head -> origin/gh/oulgen/14/head 2025-12-04T09:23:39.9698365Z * [new branch] gh/oulgen/14/orig -> origin/gh/oulgen/14/orig 2025-12-04T09:23:39.9700211Z * [new branch] gh/oulgen/15/base -> origin/gh/oulgen/15/base 2025-12-04T09:23:39.9700381Z * [new branch] gh/oulgen/15/head -> origin/gh/oulgen/15/head 2025-12-04T09:23:39.9700828Z * [new branch] gh/oulgen/15/orig -> origin/gh/oulgen/15/orig 2025-12-04T09:23:39.9701896Z * [new branch] gh/oulgen/16/base -> origin/gh/oulgen/16/base 2025-12-04T09:23:39.9702265Z * [new branch] gh/oulgen/16/head -> origin/gh/oulgen/16/head 2025-12-04T09:23:39.9703363Z * [new branch] gh/oulgen/16/orig -> origin/gh/oulgen/16/orig 2025-12-04T09:23:39.9703958Z * [new branch] gh/oulgen/17/base -> origin/gh/oulgen/17/base 2025-12-04T09:23:39.9705237Z * [new branch] gh/oulgen/17/head -> origin/gh/oulgen/17/head 2025-12-04T09:23:39.9705539Z * [new branch] gh/oulgen/17/orig -> origin/gh/oulgen/17/orig 2025-12-04T09:23:39.9706505Z * [new branch] gh/oulgen/18/base -> origin/gh/oulgen/18/base 2025-12-04T09:23:39.9707379Z * [new branch] gh/oulgen/18/head -> origin/gh/oulgen/18/head 2025-12-04T09:23:39.9707735Z * [new branch] gh/oulgen/18/orig -> origin/gh/oulgen/18/orig 2025-12-04T09:23:39.9708895Z * [new branch] gh/oulgen/19/base -> origin/gh/oulgen/19/base 2025-12-04T09:23:39.9709330Z * [new branch] gh/oulgen/19/head -> origin/gh/oulgen/19/head 2025-12-04T09:23:39.9711030Z * [new branch] gh/oulgen/19/orig -> origin/gh/oulgen/19/orig 2025-12-04T09:23:39.9711213Z * [new branch] gh/oulgen/20/base -> origin/gh/oulgen/20/base 2025-12-04T09:23:39.9711769Z * [new branch] gh/oulgen/20/head -> origin/gh/oulgen/20/head 2025-12-04T09:23:39.9712607Z * [new branch] gh/oulgen/20/orig -> origin/gh/oulgen/20/orig 2025-12-04T09:23:39.9713618Z * [new branch] gh/oulgen/21/base -> origin/gh/oulgen/21/base 2025-12-04T09:23:39.9713884Z * [new branch] gh/oulgen/21/head -> origin/gh/oulgen/21/head 2025-12-04T09:23:39.9714974Z * [new branch] gh/oulgen/21/orig -> origin/gh/oulgen/21/orig 2025-12-04T09:23:39.9715836Z * [new branch] gh/oulgen/22/base -> origin/gh/oulgen/22/base 2025-12-04T09:23:39.9716186Z * [new branch] gh/oulgen/22/head -> origin/gh/oulgen/22/head 2025-12-04T09:23:39.9717221Z * [new branch] gh/oulgen/22/orig -> origin/gh/oulgen/22/orig 2025-12-04T09:23:39.9718113Z * [new branch] gh/oulgen/23/base -> origin/gh/oulgen/23/base 2025-12-04T09:23:39.9718546Z * [new branch] gh/oulgen/23/head -> origin/gh/oulgen/23/head 2025-12-04T09:23:39.9719583Z * [new branch] gh/oulgen/23/orig -> origin/gh/oulgen/23/orig 2025-12-04T09:23:39.9720678Z * [new branch] gh/oulgen/24/base -> origin/gh/oulgen/24/base 2025-12-04T09:23:39.9721010Z * [new branch] gh/oulgen/24/head -> origin/gh/oulgen/24/head 2025-12-04T09:23:39.9722173Z * [new branch] gh/oulgen/24/orig -> origin/gh/oulgen/24/orig 2025-12-04T09:23:39.9722915Z * [new branch] gh/oulgen/25/base -> origin/gh/oulgen/25/base 2025-12-04T09:23:39.9723460Z * [new branch] gh/oulgen/25/head -> origin/gh/oulgen/25/head 2025-12-04T09:23:39.9724425Z * [new branch] gh/oulgen/25/orig -> origin/gh/oulgen/25/orig 2025-12-04T09:23:39.9725428Z * [new branch] gh/oulgen/26/base -> origin/gh/oulgen/26/base 2025-12-04T09:23:39.9725675Z * [new branch] gh/oulgen/26/head -> origin/gh/oulgen/26/head 2025-12-04T09:23:39.9726751Z * [new branch] gh/oulgen/26/orig -> origin/gh/oulgen/26/orig 2025-12-04T09:23:39.9727746Z * [new branch] gh/oulgen/4/base -> origin/gh/oulgen/4/base 2025-12-04T09:23:39.9728151Z * [new branch] gh/oulgen/4/head -> origin/gh/oulgen/4/head 2025-12-04T09:23:39.9729147Z * [new branch] gh/oulgen/4/orig -> origin/gh/oulgen/4/orig 2025-12-04T09:23:39.9730454Z * [new branch] gh/oulgen/7/base -> origin/gh/oulgen/7/base 2025-12-04T09:23:39.9731045Z * [new branch] gh/oulgen/7/head -> origin/gh/oulgen/7/head 2025-12-04T09:23:39.9732077Z * [new branch] gh/oulgen/7/orig -> origin/gh/oulgen/7/orig 2025-12-04T09:23:39.9733066Z * [new branch] gh/oulgen/8/base -> origin/gh/oulgen/8/base 2025-12-04T09:23:39.9733520Z * [new branch] gh/oulgen/8/head -> origin/gh/oulgen/8/head 2025-12-04T09:23:39.9734277Z * [new branch] gh/oulgen/8/orig -> origin/gh/oulgen/8/orig 2025-12-04T09:23:39.9735500Z * [new branch] gh/oulgen/9/base -> origin/gh/oulgen/9/base 2025-12-04T09:23:39.9735792Z * [new branch] gh/oulgen/9/head -> origin/gh/oulgen/9/head 2025-12-04T09:23:39.9736978Z * [new branch] gh/oulgen/9/orig -> origin/gh/oulgen/9/orig 2025-12-04T09:23:39.9737645Z * [new branch] gh/patvig/mtia-serialization -> origin/gh/patvig/mtia-serialization 2025-12-04T09:23:39.9739291Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-12-04T09:23:39.9739718Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-12-04T09:23:39.9740829Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-12-04T09:23:39.9749375Z * [new branch] gh/pearu/109/base -> origin/gh/pearu/109/base 2025-12-04T09:23:39.9749841Z * [new branch] gh/pearu/109/head -> origin/gh/pearu/109/head 2025-12-04T09:23:39.9750010Z * [new branch] gh/pearu/109/orig -> origin/gh/pearu/109/orig 2025-12-04T09:23:39.9750161Z * [new branch] gh/pearu/110/base -> origin/gh/pearu/110/base 2025-12-04T09:23:39.9750327Z * [new branch] gh/pearu/110/head -> origin/gh/pearu/110/head 2025-12-04T09:23:39.9750479Z * [new branch] gh/pearu/110/orig -> origin/gh/pearu/110/orig 2025-12-04T09:23:39.9750618Z * [new branch] gh/pearu/111/base -> origin/gh/pearu/111/base 2025-12-04T09:23:39.9750753Z * [new branch] gh/pearu/111/head -> origin/gh/pearu/111/head 2025-12-04T09:23:39.9750896Z * [new branch] gh/pearu/111/orig -> origin/gh/pearu/111/orig 2025-12-04T09:23:39.9751030Z * [new branch] gh/pearu/112/base -> origin/gh/pearu/112/base 2025-12-04T09:23:39.9756565Z * [new branch] gh/pearu/112/head -> origin/gh/pearu/112/head 2025-12-04T09:23:39.9756761Z * [new branch] gh/pearu/112/orig -> origin/gh/pearu/112/orig 2025-12-04T09:23:39.9756902Z * [new branch] gh/pearu/115/base -> origin/gh/pearu/115/base 2025-12-04T09:23:39.9757067Z * [new branch] gh/pearu/115/head -> origin/gh/pearu/115/head 2025-12-04T09:23:39.9757204Z * [new branch] gh/pearu/115/orig -> origin/gh/pearu/115/orig 2025-12-04T09:23:39.9757346Z * [new branch] gh/pearu/116/base -> origin/gh/pearu/116/base 2025-12-04T09:23:39.9757482Z * [new branch] gh/pearu/116/head -> origin/gh/pearu/116/head 2025-12-04T09:23:39.9757618Z * [new branch] gh/pearu/116/orig -> origin/gh/pearu/116/orig 2025-12-04T09:23:39.9757764Z * [new branch] gh/pearu/117/base -> origin/gh/pearu/117/base 2025-12-04T09:23:39.9757907Z * [new branch] gh/pearu/117/head -> origin/gh/pearu/117/head 2025-12-04T09:23:39.9758046Z * [new branch] gh/pearu/117/orig -> origin/gh/pearu/117/orig 2025-12-04T09:23:39.9758189Z * [new branch] gh/pearu/118/base -> origin/gh/pearu/118/base 2025-12-04T09:23:39.9758731Z * [new branch] gh/pearu/118/head -> origin/gh/pearu/118/head 2025-12-04T09:23:39.9759933Z * [new branch] gh/pearu/118/orig -> origin/gh/pearu/118/orig 2025-12-04T09:23:39.9760709Z * [new branch] gh/pearu/119/base -> origin/gh/pearu/119/base 2025-12-04T09:23:39.9761164Z * [new branch] gh/pearu/119/head -> origin/gh/pearu/119/head 2025-12-04T09:23:39.9762426Z * [new branch] gh/pearu/119/orig -> origin/gh/pearu/119/orig 2025-12-04T09:23:39.9763153Z * [new branch] gh/pearu/139/base -> origin/gh/pearu/139/base 2025-12-04T09:23:39.9763962Z * [new branch] gh/pearu/139/head -> origin/gh/pearu/139/head 2025-12-04T09:23:39.9764132Z * [new branch] gh/pearu/139/orig -> origin/gh/pearu/139/orig 2025-12-04T09:23:39.9769819Z * [new branch] gh/pearu/140/base -> origin/gh/pearu/140/base 2025-12-04T09:23:39.9770224Z * [new branch] gh/pearu/140/head -> origin/gh/pearu/140/head 2025-12-04T09:23:39.9770377Z * [new branch] gh/pearu/140/orig -> origin/gh/pearu/140/orig 2025-12-04T09:23:39.9770525Z * [new branch] gh/pearu/142/base -> origin/gh/pearu/142/base 2025-12-04T09:23:39.9770672Z * [new branch] gh/pearu/142/head -> origin/gh/pearu/142/head 2025-12-04T09:23:39.9770811Z * [new branch] gh/pearu/142/orig -> origin/gh/pearu/142/orig 2025-12-04T09:23:39.9770958Z * [new branch] gh/pearu/143/base -> origin/gh/pearu/143/base 2025-12-04T09:23:39.9771110Z * [new branch] gh/pearu/143/head -> origin/gh/pearu/143/head 2025-12-04T09:23:39.9771653Z * [new branch] gh/pearu/143/orig -> origin/gh/pearu/143/orig 2025-12-04T09:23:39.9772990Z * [new branch] gh/pearu/147/base -> origin/gh/pearu/147/base 2025-12-04T09:23:39.9773236Z * [new branch] gh/pearu/147/head -> origin/gh/pearu/147/head 2025-12-04T09:23:39.9775435Z * [new branch] gh/pearu/147/orig -> origin/gh/pearu/147/orig 2025-12-04T09:23:39.9775610Z * [new branch] gh/pearu/149/base -> origin/gh/pearu/149/base 2025-12-04T09:23:39.9775753Z * [new branch] gh/pearu/149/head -> origin/gh/pearu/149/head 2025-12-04T09:23:39.9776298Z * [new branch] gh/pearu/149/orig -> origin/gh/pearu/149/orig 2025-12-04T09:23:39.9780406Z * [new branch] gh/pearu/150/base -> origin/gh/pearu/150/base 2025-12-04T09:23:39.9780608Z * [new branch] gh/pearu/150/head -> origin/gh/pearu/150/head 2025-12-04T09:23:39.9780746Z * [new branch] gh/pearu/150/orig -> origin/gh/pearu/150/orig 2025-12-04T09:23:39.9785313Z * [new branch] gh/pearu/151/base -> origin/gh/pearu/151/base 2025-12-04T09:23:39.9785504Z * [new branch] gh/pearu/151/head -> origin/gh/pearu/151/head 2025-12-04T09:23:39.9785644Z * [new branch] gh/pearu/151/orig -> origin/gh/pearu/151/orig 2025-12-04T09:23:39.9785781Z * [new branch] gh/pearu/152/base -> origin/gh/pearu/152/base 2025-12-04T09:23:39.9785917Z * [new branch] gh/pearu/152/head -> origin/gh/pearu/152/head 2025-12-04T09:23:39.9788242Z * [new branch] gh/pearu/152/orig -> origin/gh/pearu/152/orig 2025-12-04T09:23:39.9788426Z * [new branch] gh/pearu/153/base -> origin/gh/pearu/153/base 2025-12-04T09:23:39.9788603Z * [new branch] gh/pearu/153/head -> origin/gh/pearu/153/head 2025-12-04T09:23:39.9788742Z * [new branch] gh/pearu/153/orig -> origin/gh/pearu/153/orig 2025-12-04T09:23:39.9788879Z * [new branch] gh/pearu/154/base -> origin/gh/pearu/154/base 2025-12-04T09:23:39.9789044Z * [new branch] gh/pearu/154/head -> origin/gh/pearu/154/head 2025-12-04T09:23:39.9789179Z * [new branch] gh/pearu/154/orig -> origin/gh/pearu/154/orig 2025-12-04T09:23:39.9789316Z * [new branch] gh/pearu/155/base -> origin/gh/pearu/155/base 2025-12-04T09:23:39.9794417Z * [new branch] gh/pearu/155/head -> origin/gh/pearu/155/head 2025-12-04T09:23:39.9794603Z * [new branch] gh/pearu/155/orig -> origin/gh/pearu/155/orig 2025-12-04T09:23:39.9794757Z * [new branch] gh/pearu/156/base -> origin/gh/pearu/156/base 2025-12-04T09:23:39.9795058Z * [new branch] gh/pearu/156/head -> origin/gh/pearu/156/head 2025-12-04T09:23:39.9795206Z * [new branch] gh/pearu/156/orig -> origin/gh/pearu/156/orig 2025-12-04T09:23:39.9795354Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-12-04T09:23:39.9795549Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-12-04T09:23:39.9795693Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-12-04T09:23:39.9796086Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-12-04T09:23:39.9796251Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-12-04T09:23:39.9796842Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-12-04T09:23:39.9798382Z * [new branch] gh/pianpwk/21/base -> origin/gh/pianpwk/21/base 2025-12-04T09:23:39.9798922Z * [new branch] gh/pianpwk/21/head -> origin/gh/pianpwk/21/head 2025-12-04T09:23:39.9800161Z * [new branch] gh/pianpwk/28/base -> origin/gh/pianpwk/28/base 2025-12-04T09:23:39.9800827Z * [new branch] gh/pianpwk/28/head -> origin/gh/pianpwk/28/head 2025-12-04T09:23:39.9801875Z * [new branch] gh/pianpwk/28/orig -> origin/gh/pianpwk/28/orig 2025-12-04T09:23:39.9802660Z * [new branch] gh/pianpwk/29/base -> origin/gh/pianpwk/29/base 2025-12-04T09:23:39.9803524Z * [new branch] gh/pianpwk/29/head -> origin/gh/pianpwk/29/head 2025-12-04T09:23:39.9804733Z * [new branch] gh/pianpwk/29/orig -> origin/gh/pianpwk/29/orig 2025-12-04T09:23:39.9805847Z * [new branch] gh/pianpwk/30/base -> origin/gh/pianpwk/30/base 2025-12-04T09:23:39.9806342Z * [new branch] gh/pianpwk/30/head -> origin/gh/pianpwk/30/head 2025-12-04T09:23:39.9807216Z * [new branch] gh/pianpwk/30/orig -> origin/gh/pianpwk/30/orig 2025-12-04T09:23:39.9808305Z * [new branch] gh/pianpwk/31/base -> origin/gh/pianpwk/31/base 2025-12-04T09:23:39.9809331Z * [new branch] gh/pianpwk/31/head -> origin/gh/pianpwk/31/head 2025-12-04T09:23:39.9809599Z * [new branch] gh/pianpwk/31/orig -> origin/gh/pianpwk/31/orig 2025-12-04T09:23:39.9810821Z * [new branch] gh/pianpwk/32/base -> origin/gh/pianpwk/32/base 2025-12-04T09:23:39.9811070Z * [new branch] gh/pianpwk/32/head -> origin/gh/pianpwk/32/head 2025-12-04T09:23:39.9812126Z * [new branch] gh/pianpwk/32/orig -> origin/gh/pianpwk/32/orig 2025-12-04T09:23:39.9812855Z * [new branch] gh/pianpwk/33/base -> origin/gh/pianpwk/33/base 2025-12-04T09:23:39.9813482Z * [new branch] gh/pianpwk/33/head -> origin/gh/pianpwk/33/head 2025-12-04T09:23:39.9814271Z * [new branch] gh/pianpwk/33/orig -> origin/gh/pianpwk/33/orig 2025-12-04T09:23:39.9816054Z * [new branch] gh/pianpwk/34/base -> origin/gh/pianpwk/34/base 2025-12-04T09:23:39.9816211Z * [new branch] gh/pianpwk/34/head -> origin/gh/pianpwk/34/head 2025-12-04T09:23:39.9820002Z * [new branch] gh/pianpwk/34/orig -> origin/gh/pianpwk/34/orig 2025-12-04T09:23:39.9820145Z * [new branch] gh/pianpwk/35/base -> origin/gh/pianpwk/35/base 2025-12-04T09:23:39.9820289Z * [new branch] gh/pianpwk/35/head -> origin/gh/pianpwk/35/head 2025-12-04T09:23:39.9820434Z * [new branch] gh/pianpwk/35/orig -> origin/gh/pianpwk/35/orig 2025-12-04T09:23:39.9820837Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-12-04T09:23:39.9821787Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-12-04T09:23:39.9822454Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-12-04T09:23:39.9823354Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-12-04T09:23:39.9823798Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-12-04T09:23:39.9825632Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-12-04T09:23:39.9825883Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-12-04T09:23:39.9826017Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-12-04T09:23:39.9827459Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-12-04T09:23:39.9827718Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-12-04T09:23:39.9828724Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-12-04T09:23:39.9829650Z * [new branch] gh/rec/166/base -> origin/gh/rec/166/base 2025-12-04T09:23:39.9830102Z * [new branch] gh/rec/166/head -> origin/gh/rec/166/head 2025-12-04T09:23:39.9831167Z * [new branch] gh/rec/166/orig -> origin/gh/rec/166/orig 2025-12-04T09:23:39.9831767Z * [new branch] gh/rec/167/base -> origin/gh/rec/167/base 2025-12-04T09:23:39.9832702Z * [new branch] gh/rec/167/head -> origin/gh/rec/167/head 2025-12-04T09:23:39.9833259Z * [new branch] gh/rec/167/orig -> origin/gh/rec/167/orig 2025-12-04T09:23:39.9835011Z * [new branch] gh/rec/168/base -> origin/gh/rec/168/base 2025-12-04T09:23:39.9835191Z * [new branch] gh/rec/168/head -> origin/gh/rec/168/head 2025-12-04T09:23:39.9835382Z * [new branch] gh/rec/168/orig -> origin/gh/rec/168/orig 2025-12-04T09:23:39.9837254Z * [new branch] gh/rec/169/base -> origin/gh/rec/169/base 2025-12-04T09:23:39.9837432Z * [new branch] gh/rec/169/head -> origin/gh/rec/169/head 2025-12-04T09:23:39.9837940Z * [new branch] gh/rec/169/orig -> origin/gh/rec/169/orig 2025-12-04T09:23:39.9839236Z * [new branch] gh/rec/170/base -> origin/gh/rec/170/base 2025-12-04T09:23:39.9839451Z * [new branch] gh/rec/170/head -> origin/gh/rec/170/head 2025-12-04T09:23:39.9840810Z * [new branch] gh/rec/170/orig -> origin/gh/rec/170/orig 2025-12-04T09:23:39.9841457Z * [new branch] gh/rec/171/base -> origin/gh/rec/171/base 2025-12-04T09:23:39.9845812Z * [new branch] gh/rec/171/head -> origin/gh/rec/171/head 2025-12-04T09:23:39.9845987Z * [new branch] gh/rec/171/orig -> origin/gh/rec/171/orig 2025-12-04T09:23:39.9846160Z * [new branch] gh/rec/172/base -> origin/gh/rec/172/base 2025-12-04T09:23:39.9846298Z * [new branch] gh/rec/172/head -> origin/gh/rec/172/head 2025-12-04T09:23:39.9846441Z * [new branch] gh/rec/172/orig -> origin/gh/rec/172/orig 2025-12-04T09:23:39.9846590Z * [new branch] gh/rec/173/base -> origin/gh/rec/173/base 2025-12-04T09:23:39.9847351Z * [new branch] gh/rec/173/head -> origin/gh/rec/173/head 2025-12-04T09:23:39.9854103Z * [new branch] gh/rec/173/orig -> origin/gh/rec/173/orig 2025-12-04T09:23:39.9854274Z * [new branch] gh/rec/174/base -> origin/gh/rec/174/base 2025-12-04T09:23:39.9854416Z * [new branch] gh/rec/174/head -> origin/gh/rec/174/head 2025-12-04T09:23:39.9858468Z * [new branch] gh/rec/174/orig -> origin/gh/rec/174/orig 2025-12-04T09:23:39.9858930Z * [new branch] gh/rec/175/base -> origin/gh/rec/175/base 2025-12-04T09:23:39.9859092Z * [new branch] gh/rec/175/head -> origin/gh/rec/175/head 2025-12-04T09:23:39.9859254Z * [new branch] gh/rec/175/orig -> origin/gh/rec/175/orig 2025-12-04T09:23:39.9859471Z * [new branch] gh/rec/176/base -> origin/gh/rec/176/base 2025-12-04T09:23:39.9864316Z * [new branch] gh/rec/176/head -> origin/gh/rec/176/head 2025-12-04T09:23:39.9869467Z * [new branch] gh/rec/176/orig -> origin/gh/rec/176/orig 2025-12-04T09:23:39.9874317Z * [new branch] gh/rec/177/base -> origin/gh/rec/177/base 2025-12-04T09:23:39.9876778Z * [new branch] gh/rec/177/head -> origin/gh/rec/177/head 2025-12-04T09:23:39.9876963Z * [new branch] gh/rec/177/orig -> origin/gh/rec/177/orig 2025-12-04T09:23:39.9877173Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-12-04T09:23:39.9877361Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-12-04T09:23:39.9877525Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-12-04T09:23:39.9877704Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-12-04T09:23:39.9877867Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-12-04T09:23:39.9878028Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-12-04T09:23:39.9878194Z * [new branch] gh/robert-hardwick/5/base -> origin/gh/robert-hardwick/5/base 2025-12-04T09:23:39.9878354Z * [new branch] gh/robert-hardwick/5/head -> origin/gh/robert-hardwick/5/head 2025-12-04T09:23:39.9878520Z * [new branch] gh/robert-hardwick/5/orig -> origin/gh/robert-hardwick/5/orig 2025-12-04T09:23:39.9878691Z * [new branch] gh/robert-hardwick/6/base -> origin/gh/robert-hardwick/6/base 2025-12-04T09:23:39.9878856Z * [new branch] gh/robert-hardwick/6/head -> origin/gh/robert-hardwick/6/head 2025-12-04T09:23:39.9879026Z * [new branch] gh/robert-hardwick/6/orig -> origin/gh/robert-hardwick/6/orig 2025-12-04T09:23:39.9879190Z * [new branch] gh/robert-hardwick/7/base -> origin/gh/robert-hardwick/7/base 2025-12-04T09:23:39.9879356Z * [new branch] gh/robert-hardwick/7/head -> origin/gh/robert-hardwick/7/head 2025-12-04T09:23:39.9879515Z * [new branch] gh/robert-hardwick/7/orig -> origin/gh/robert-hardwick/7/orig 2025-12-04T09:23:39.9879673Z * [new branch] gh/robert-hardwick/8/base -> origin/gh/robert-hardwick/8/base 2025-12-04T09:23:39.9879836Z * [new branch] gh/robert-hardwick/8/head -> origin/gh/robert-hardwick/8/head 2025-12-04T09:23:39.9879997Z * [new branch] gh/robert-hardwick/8/orig -> origin/gh/robert-hardwick/8/orig 2025-12-04T09:23:39.9880156Z * [new branch] gh/robert-hardwick/9/base -> origin/gh/robert-hardwick/9/base 2025-12-04T09:23:39.9880683Z * [new branch] gh/robert-hardwick/9/head -> origin/gh/robert-hardwick/9/head 2025-12-04T09:23:39.9880884Z * [new branch] gh/robert-hardwick/9/orig -> origin/gh/robert-hardwick/9/orig 2025-12-04T09:23:39.9885822Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-12-04T09:23:39.9889128Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-12-04T09:23:39.9889292Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-12-04T09:23:39.9889430Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-12-04T09:23:39.9889591Z * [new branch] gh/rtimpe/22/base -> origin/gh/rtimpe/22/base 2025-12-04T09:23:39.9889887Z * [new branch] gh/rtimpe/22/head -> origin/gh/rtimpe/22/head 2025-12-04T09:23:39.9890035Z * [new branch] gh/rtimpe/22/orig -> origin/gh/rtimpe/22/orig 2025-12-04T09:23:39.9890184Z * [new branch] gh/rtimpe/23/base -> origin/gh/rtimpe/23/base 2025-12-04T09:23:39.9890393Z * [new branch] gh/rtimpe/23/head -> origin/gh/rtimpe/23/head 2025-12-04T09:23:39.9890541Z * [new branch] gh/rtimpe/23/orig -> origin/gh/rtimpe/23/orig 2025-12-04T09:23:39.9896179Z * [new branch] gh/rtimpe/24/base -> origin/gh/rtimpe/24/base 2025-12-04T09:23:39.9901153Z * [new branch] gh/rtimpe/24/head -> origin/gh/rtimpe/24/head 2025-12-04T09:23:39.9904834Z * [new branch] gh/rtimpe/24/orig -> origin/gh/rtimpe/24/orig 2025-12-04T09:23:39.9909507Z * [new branch] gh/rtimpe/25/base -> origin/gh/rtimpe/25/base 2025-12-04T09:23:39.9911602Z * [new branch] gh/rtimpe/25/head -> origin/gh/rtimpe/25/head 2025-12-04T09:23:39.9911945Z * [new branch] gh/rtimpe/25/orig -> origin/gh/rtimpe/25/orig 2025-12-04T09:23:39.9912124Z * [new branch] gh/rtimpe/26/base -> origin/gh/rtimpe/26/base 2025-12-04T09:23:39.9912417Z * [new branch] gh/rtimpe/26/head -> origin/gh/rtimpe/26/head 2025-12-04T09:23:39.9912573Z * [new branch] gh/rtimpe/26/orig -> origin/gh/rtimpe/26/orig 2025-12-04T09:23:39.9912724Z * [new branch] gh/rtimpe/27/base -> origin/gh/rtimpe/27/base 2025-12-04T09:23:39.9913001Z * [new branch] gh/rtimpe/27/head -> origin/gh/rtimpe/27/head 2025-12-04T09:23:39.9913166Z * [new branch] gh/rtimpe/27/orig -> origin/gh/rtimpe/27/orig 2025-12-04T09:23:39.9913308Z * [new branch] gh/rtimpe/28/base -> origin/gh/rtimpe/28/base 2025-12-04T09:23:39.9913632Z * [new branch] gh/rtimpe/28/head -> origin/gh/rtimpe/28/head 2025-12-04T09:23:39.9913822Z * [new branch] gh/rtimpe/28/orig -> origin/gh/rtimpe/28/orig 2025-12-04T09:23:39.9913983Z * [new branch] gh/rtimpe/29/base -> origin/gh/rtimpe/29/base 2025-12-04T09:23:39.9914147Z * [new branch] gh/rtimpe/29/head -> origin/gh/rtimpe/29/head 2025-12-04T09:23:39.9914287Z * [new branch] gh/rtimpe/29/orig -> origin/gh/rtimpe/29/orig 2025-12-04T09:23:39.9914469Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-12-04T09:23:39.9914617Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-12-04T09:23:39.9914760Z * [new branch] gh/rtimpe/30/base -> origin/gh/rtimpe/30/base 2025-12-04T09:23:39.9914906Z * [new branch] gh/rtimpe/30/head -> origin/gh/rtimpe/30/head 2025-12-04T09:23:39.9915052Z * [new branch] gh/rtimpe/30/orig -> origin/gh/rtimpe/30/orig 2025-12-04T09:23:39.9915193Z * [new branch] gh/rtimpe/31/base -> origin/gh/rtimpe/31/base 2025-12-04T09:23:39.9915340Z * [new branch] gh/rtimpe/31/head -> origin/gh/rtimpe/31/head 2025-12-04T09:23:39.9915485Z * [new branch] gh/rtimpe/31/orig -> origin/gh/rtimpe/31/orig 2025-12-04T09:23:39.9915628Z * [new branch] gh/rtimpe/32/base -> origin/gh/rtimpe/32/base 2025-12-04T09:23:39.9915768Z * [new branch] gh/rtimpe/32/head -> origin/gh/rtimpe/32/head 2025-12-04T09:23:39.9915907Z * [new branch] gh/rtimpe/32/orig -> origin/gh/rtimpe/32/orig 2025-12-04T09:23:39.9916054Z * [new branch] gh/rtimpe/33/base -> origin/gh/rtimpe/33/base 2025-12-04T09:23:39.9916194Z * [new branch] gh/rtimpe/33/head -> origin/gh/rtimpe/33/head 2025-12-04T09:23:39.9916731Z * [new branch] gh/rtimpe/33/orig -> origin/gh/rtimpe/33/orig 2025-12-04T09:23:39.9916870Z * [new branch] gh/rtimpe/34/base -> origin/gh/rtimpe/34/base 2025-12-04T09:23:39.9917014Z * [new branch] gh/rtimpe/34/head -> origin/gh/rtimpe/34/head 2025-12-04T09:23:39.9917222Z * [new branch] gh/rtimpe/34/orig -> origin/gh/rtimpe/34/orig 2025-12-04T09:23:39.9917366Z * [new branch] gh/rtimpe/35/base -> origin/gh/rtimpe/35/base 2025-12-04T09:23:39.9917515Z * [new branch] gh/rtimpe/35/head -> origin/gh/rtimpe/35/head 2025-12-04T09:23:39.9917993Z * [new branch] gh/rtimpe/35/orig -> origin/gh/rtimpe/35/orig 2025-12-04T09:23:39.9918578Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-12-04T09:23:39.9918748Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-12-04T09:23:39.9918949Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-12-04T09:23:39.9919123Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-12-04T09:23:39.9919287Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-12-04T09:23:39.9920002Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-12-04T09:23:39.9921161Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-12-04T09:23:39.9924657Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-12-04T09:23:39.9925065Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-12-04T09:23:39.9925329Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-12-04T09:23:39.9925599Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-12-04T09:23:39.9925842Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-12-04T09:23:39.9930531Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-12-04T09:23:39.9930764Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-12-04T09:23:39.9930926Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-12-04T09:23:39.9931170Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-12-04T09:23:39.9931365Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-12-04T09:23:39.9932896Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-12-04T09:23:39.9933111Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-12-04T09:23:39.9933314Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-12-04T09:23:39.9933491Z * [new branch] gh/ruisizhang123/9/base -> origin/gh/ruisizhang123/9/base 2025-12-04T09:23:39.9933654Z * [new branch] gh/ruisizhang123/9/head -> origin/gh/ruisizhang123/9/head 2025-12-04T09:23:39.9933824Z * [new branch] gh/ruisizhang123/9/orig -> origin/gh/ruisizhang123/9/orig 2025-12-04T09:23:39.9940041Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-12-04T09:23:39.9940640Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-12-04T09:23:39.9940842Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-12-04T09:23:39.9941009Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-12-04T09:23:39.9941170Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-12-04T09:23:39.9941501Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-12-04T09:23:39.9941686Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-12-04T09:23:39.9941877Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-12-04T09:23:39.9942207Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-12-04T09:23:39.9942371Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-12-04T09:23:39.9942523Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-12-04T09:23:39.9942701Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-12-04T09:23:39.9949656Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-12-04T09:23:39.9952168Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-12-04T09:23:39.9952362Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-12-04T09:23:39.9952523Z * [new branch] gh/seemethere/62/base -> origin/gh/seemethere/62/base 2025-12-04T09:23:39.9952707Z * [new branch] gh/seemethere/62/head -> origin/gh/seemethere/62/head 2025-12-04T09:23:39.9952864Z * [new branch] gh/seemethere/62/orig -> origin/gh/seemethere/62/orig 2025-12-04T09:23:39.9953019Z * [new branch] gh/seemethere/63/base -> origin/gh/seemethere/63/base 2025-12-04T09:23:39.9953173Z * [new branch] gh/seemethere/63/head -> origin/gh/seemethere/63/head 2025-12-04T09:23:39.9953336Z * [new branch] gh/seemethere/63/orig -> origin/gh/seemethere/63/orig 2025-12-04T09:23:39.9953492Z * [new branch] gh/seemethere/71/base -> origin/gh/seemethere/71/base 2025-12-04T09:23:39.9956342Z * [new branch] gh/seemethere/71/head -> origin/gh/seemethere/71/head 2025-12-04T09:23:39.9956970Z * [new branch] gh/seemethere/71/orig -> origin/gh/seemethere/71/orig 2025-12-04T09:23:39.9957157Z * [new branch] gh/seemethere/72/base -> origin/gh/seemethere/72/base 2025-12-04T09:23:39.9957334Z * [new branch] gh/seemethere/72/head -> origin/gh/seemethere/72/head 2025-12-04T09:23:39.9957496Z * [new branch] gh/seemethere/72/orig -> origin/gh/seemethere/72/orig 2025-12-04T09:23:39.9957641Z * [new branch] gh/seemethere/73/base -> origin/gh/seemethere/73/base 2025-12-04T09:23:39.9957797Z * [new branch] gh/seemethere/73/head -> origin/gh/seemethere/73/head 2025-12-04T09:23:39.9957947Z * [new branch] gh/seemethere/73/orig -> origin/gh/seemethere/73/orig 2025-12-04T09:23:39.9960469Z * [new branch] gh/seemethere/74/base -> origin/gh/seemethere/74/base 2025-12-04T09:23:39.9960665Z * [new branch] gh/seemethere/74/head -> origin/gh/seemethere/74/head 2025-12-04T09:23:39.9961233Z * [new branch] gh/seemethere/74/orig -> origin/gh/seemethere/74/orig 2025-12-04T09:23:39.9961426Z * [new branch] gh/seemethere/75/base -> origin/gh/seemethere/75/base 2025-12-04T09:23:39.9961644Z * [new branch] gh/seemethere/75/head -> origin/gh/seemethere/75/head 2025-12-04T09:23:39.9961811Z * [new branch] gh/seemethere/75/orig -> origin/gh/seemethere/75/orig 2025-12-04T09:23:39.9964362Z * [new branch] gh/seemethere/76/base -> origin/gh/seemethere/76/base 2025-12-04T09:23:39.9964517Z * [new branch] gh/seemethere/76/head -> origin/gh/seemethere/76/head 2025-12-04T09:23:39.9964723Z * [new branch] gh/seemethere/76/orig -> origin/gh/seemethere/76/orig 2025-12-04T09:23:39.9971572Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-12-04T09:23:39.9976760Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-12-04T09:23:39.9980734Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-12-04T09:23:39.9985309Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-12-04T09:23:39.9985524Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-12-04T09:23:39.9985681Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-12-04T09:23:39.9985827Z * [new branch] gh/shunting314/249/base -> origin/gh/shunting314/249/base 2025-12-04T09:23:39.9985971Z * [new branch] gh/shunting314/249/head -> origin/gh/shunting314/249/head 2025-12-04T09:23:39.9986119Z * [new branch] gh/shunting314/249/orig -> origin/gh/shunting314/249/orig 2025-12-04T09:23:39.9986276Z * [new branch] gh/shunting314/253/base -> origin/gh/shunting314/253/base 2025-12-04T09:23:39.9986425Z * [new branch] gh/shunting314/253/head -> origin/gh/shunting314/253/head 2025-12-04T09:23:39.9986583Z * [new branch] gh/shunting314/253/orig -> origin/gh/shunting314/253/orig 2025-12-04T09:23:39.9986734Z * [new branch] gh/shunting314/256/base -> origin/gh/shunting314/256/base 2025-12-04T09:23:39.9986889Z * [new branch] gh/shunting314/256/head -> origin/gh/shunting314/256/head 2025-12-04T09:23:39.9987038Z * [new branch] gh/shunting314/256/orig -> origin/gh/shunting314/256/orig 2025-12-04T09:23:39.9987186Z * [new branch] gh/shunting314/257/base -> origin/gh/shunting314/257/base 2025-12-04T09:23:39.9987342Z * [new branch] gh/shunting314/257/head -> origin/gh/shunting314/257/head 2025-12-04T09:23:39.9987495Z * [new branch] gh/shunting314/257/orig -> origin/gh/shunting314/257/orig 2025-12-04T09:23:39.9987648Z * [new branch] gh/shunting314/258/base -> origin/gh/shunting314/258/base 2025-12-04T09:23:39.9987794Z * [new branch] gh/shunting314/258/head -> origin/gh/shunting314/258/head 2025-12-04T09:23:39.9987944Z * [new branch] gh/shunting314/258/orig -> origin/gh/shunting314/258/orig 2025-12-04T09:23:39.9988098Z * [new branch] gh/shunting314/259/base -> origin/gh/shunting314/259/base 2025-12-04T09:23:39.9988245Z * [new branch] gh/shunting314/259/head -> origin/gh/shunting314/259/head 2025-12-04T09:23:39.9988407Z * [new branch] gh/shunting314/259/orig -> origin/gh/shunting314/259/orig 2025-12-04T09:23:39.9988556Z * [new branch] gh/shunting314/260/base -> origin/gh/shunting314/260/base 2025-12-04T09:23:39.9988702Z * [new branch] gh/shunting314/260/head -> origin/gh/shunting314/260/head 2025-12-04T09:23:39.9988863Z * [new branch] gh/shunting314/260/orig -> origin/gh/shunting314/260/orig 2025-12-04T09:23:39.9989010Z * [new branch] gh/shunting314/261/base -> origin/gh/shunting314/261/base 2025-12-04T09:23:39.9989162Z * [new branch] gh/shunting314/261/head -> origin/gh/shunting314/261/head 2025-12-04T09:23:39.9989319Z * [new branch] gh/shunting314/261/orig -> origin/gh/shunting314/261/orig 2025-12-04T09:23:39.9990005Z * [new branch] gh/shunting314/262/base -> origin/gh/shunting314/262/base 2025-12-04T09:23:39.9990723Z * [new branch] gh/shunting314/262/head -> origin/gh/shunting314/262/head 2025-12-04T09:23:39.9991724Z * [new branch] gh/shunting314/262/orig -> origin/gh/shunting314/262/orig 2025-12-04T09:23:39.9997826Z * [new branch] gh/shunting314/263/base -> origin/gh/shunting314/263/base 2025-12-04T09:23:39.9998173Z * [new branch] gh/shunting314/263/head -> origin/gh/shunting314/263/head 2025-12-04T09:23:39.9998329Z * [new branch] gh/shunting314/263/orig -> origin/gh/shunting314/263/orig 2025-12-04T09:23:39.9998489Z * [new branch] gh/shunting314/264/base -> origin/gh/shunting314/264/base 2025-12-04T09:23:39.9998697Z * [new branch] gh/shunting314/264/head -> origin/gh/shunting314/264/head 2025-12-04T09:23:39.9998858Z * [new branch] gh/shunting314/264/orig -> origin/gh/shunting314/264/orig 2025-12-04T09:23:39.9999009Z * [new branch] gh/shunting314/265/base -> origin/gh/shunting314/265/base 2025-12-04T09:23:39.9999160Z * [new branch] gh/shunting314/265/head -> origin/gh/shunting314/265/head 2025-12-04T09:23:39.9999317Z * [new branch] gh/shunting314/265/orig -> origin/gh/shunting314/265/orig 2025-12-04T09:23:39.9999891Z * [new branch] gh/shunting314/266/base -> origin/gh/shunting314/266/base 2025-12-04T09:23:40.0000938Z * [new branch] gh/shunting314/266/head -> origin/gh/shunting314/266/head 2025-12-04T09:23:40.0007250Z * [new branch] gh/shunting314/266/orig -> origin/gh/shunting314/266/orig 2025-12-04T09:23:40.0007452Z * [new branch] gh/shunting314/267/base -> origin/gh/shunting314/267/base 2025-12-04T09:23:40.0007629Z * [new branch] gh/shunting314/267/head -> origin/gh/shunting314/267/head 2025-12-04T09:23:40.0007782Z * [new branch] gh/shunting314/267/orig -> origin/gh/shunting314/267/orig 2025-12-04T09:23:40.0009691Z * [new branch] gh/shunting314/268/base -> origin/gh/shunting314/268/base 2025-12-04T09:23:40.0010059Z * [new branch] gh/shunting314/268/head -> origin/gh/shunting314/268/head 2025-12-04T09:23:40.0010268Z * [new branch] gh/shunting314/268/orig -> origin/gh/shunting314/268/orig 2025-12-04T09:23:40.0010531Z * [new branch] gh/shunting314/269/base -> origin/gh/shunting314/269/base 2025-12-04T09:23:40.0010800Z * [new branch] gh/shunting314/269/head -> origin/gh/shunting314/269/head 2025-12-04T09:23:40.0011047Z * [new branch] gh/shunting314/269/orig -> origin/gh/shunting314/269/orig 2025-12-04T09:23:40.0011316Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-12-04T09:23:40.0014622Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-12-04T09:23:40.0014801Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-12-04T09:23:40.0014952Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-12-04T09:23:40.0015229Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-12-04T09:23:40.0015396Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-12-04T09:23:40.0015642Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-12-04T09:23:40.0017717Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-12-04T09:23:40.0018289Z * [new branch] gh/slayton58/39/base -> origin/gh/slayton58/39/base 2025-12-04T09:23:40.0018490Z * [new branch] gh/slayton58/39/head -> origin/gh/slayton58/39/head 2025-12-04T09:23:40.0018653Z * [new branch] gh/slayton58/39/orig -> origin/gh/slayton58/39/orig 2025-12-04T09:23:40.0020891Z * [new branch] gh/slayton58/42/base -> origin/gh/slayton58/42/base 2025-12-04T09:23:40.0021048Z * [new branch] gh/slayton58/42/head -> origin/gh/slayton58/42/head 2025-12-04T09:23:40.0021203Z * [new branch] gh/slayton58/42/orig -> origin/gh/slayton58/42/orig 2025-12-04T09:23:40.0021354Z * [new branch] gh/slayton58/43/base -> origin/gh/slayton58/43/base 2025-12-04T09:23:40.0021645Z * [new branch] gh/slayton58/43/head -> origin/gh/slayton58/43/head 2025-12-04T09:23:40.0022267Z * [new branch] gh/slayton58/43/orig -> origin/gh/slayton58/43/orig 2025-12-04T09:23:40.0023670Z * [new branch] gh/slayton58/44/base -> origin/gh/slayton58/44/base 2025-12-04T09:23:40.0023989Z * [new branch] gh/slayton58/44/head -> origin/gh/slayton58/44/head 2025-12-04T09:23:40.0025100Z * [new branch] gh/slayton58/44/orig -> origin/gh/slayton58/44/orig 2025-12-04T09:23:40.0025959Z * [new branch] gh/slayton58/45/base -> origin/gh/slayton58/45/base 2025-12-04T09:23:40.0026452Z * [new branch] gh/slayton58/45/head -> origin/gh/slayton58/45/head 2025-12-04T09:23:40.0027467Z * [new branch] gh/slayton58/45/orig -> origin/gh/slayton58/45/orig 2025-12-04T09:23:40.0028470Z * [new branch] gh/slayton58/46/base -> origin/gh/slayton58/46/base 2025-12-04T09:23:40.0029030Z * [new branch] gh/slayton58/46/head -> origin/gh/slayton58/46/head 2025-12-04T09:23:40.0030439Z * [new branch] gh/slayton58/46/orig -> origin/gh/slayton58/46/orig 2025-12-04T09:23:40.0030760Z * [new branch] gh/slayton58/6/base -> origin/gh/slayton58/6/base 2025-12-04T09:23:40.0032696Z * [new branch] gh/slayton58/6/head -> origin/gh/slayton58/6/head 2025-12-04T09:23:40.0032844Z * [new branch] gh/slayton58/7/base -> origin/gh/slayton58/7/base 2025-12-04T09:23:40.0037044Z * [new branch] gh/slayton58/7/head -> origin/gh/slayton58/7/head 2025-12-04T09:23:40.0037234Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-12-04T09:23:40.0037387Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-12-04T09:23:40.0037558Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-12-04T09:23:40.0037703Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-12-04T09:23:40.0037871Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-12-04T09:23:40.0040908Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-12-04T09:23:40.0041066Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-12-04T09:23:40.0041219Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-12-04T09:23:40.0041372Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-12-04T09:23:40.0047995Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-12-04T09:23:40.0051261Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-12-04T09:23:40.0051450Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-12-04T09:23:40.0051615Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-12-04T09:23:40.0051771Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-12-04T09:23:40.0051943Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-12-04T09:23:40.0052115Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-12-04T09:23:40.0052269Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-12-04T09:23:40.0052428Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-12-04T09:23:40.0052581Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-12-04T09:23:40.0054800Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-12-04T09:23:40.0055243Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-12-04T09:23:40.0058790Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-12-04T09:23:40.0059099Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-12-04T09:23:40.0059340Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-12-04T09:23:40.0059496Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-12-04T09:23:40.0059646Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-12-04T09:23:40.0059803Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-12-04T09:23:40.0059964Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-12-04T09:23:40.0060147Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-12-04T09:23:40.0060298Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-12-04T09:23:40.0060441Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-12-04T09:23:40.0060594Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-12-04T09:23:40.0065705Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-12-04T09:23:40.0070129Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-12-04T09:23:40.0074494Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-12-04T09:23:40.0074706Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-12-04T09:23:40.0074871Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-12-04T09:23:40.0075046Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-12-04T09:23:40.0075206Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-12-04T09:23:40.0075366Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-12-04T09:23:40.0075543Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-12-04T09:23:40.0075692Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-12-04T09:23:40.0075852Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-12-04T09:23:40.0076005Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-12-04T09:23:40.0076195Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-12-04T09:23:40.0076357Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-12-04T09:23:40.0076513Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-12-04T09:23:40.0076668Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-12-04T09:23:40.0076831Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-12-04T09:23:40.0076991Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-12-04T09:23:40.0077152Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-12-04T09:23:40.0077307Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-12-04T09:23:40.0077522Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-12-04T09:23:40.0078218Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-12-04T09:23:40.0079687Z * [new branch] gh/soulitzer/374/base -> origin/gh/soulitzer/374/base 2025-12-04T09:23:40.0080215Z * [new branch] gh/soulitzer/374/head -> origin/gh/soulitzer/374/head 2025-12-04T09:23:40.0081079Z * [new branch] gh/soulitzer/374/orig -> origin/gh/soulitzer/374/orig 2025-12-04T09:23:40.0087307Z * [new branch] gh/soulitzer/375/base -> origin/gh/soulitzer/375/base 2025-12-04T09:23:40.0087764Z * [new branch] gh/soulitzer/375/head -> origin/gh/soulitzer/375/head 2025-12-04T09:23:40.0087939Z * [new branch] gh/soulitzer/375/orig -> origin/gh/soulitzer/375/orig 2025-12-04T09:23:40.0088099Z * [new branch] gh/soulitzer/380/base -> origin/gh/soulitzer/380/base 2025-12-04T09:23:40.0088251Z * [new branch] gh/soulitzer/380/head -> origin/gh/soulitzer/380/head 2025-12-04T09:23:40.0088408Z * [new branch] gh/soulitzer/380/orig -> origin/gh/soulitzer/380/orig 2025-12-04T09:23:40.0088573Z * [new branch] gh/soulitzer/385/base -> origin/gh/soulitzer/385/base 2025-12-04T09:23:40.0088718Z * [new branch] gh/soulitzer/385/head -> origin/gh/soulitzer/385/head 2025-12-04T09:23:40.0088875Z * [new branch] gh/soulitzer/385/orig -> origin/gh/soulitzer/385/orig 2025-12-04T09:23:40.0094134Z * [new branch] gh/soulitzer/386/base -> origin/gh/soulitzer/386/base 2025-12-04T09:23:40.0094468Z * [new branch] gh/soulitzer/386/head -> origin/gh/soulitzer/386/head 2025-12-04T09:23:40.0094625Z * [new branch] gh/soulitzer/386/orig -> origin/gh/soulitzer/386/orig 2025-12-04T09:23:40.0094781Z * [new branch] gh/soulitzer/387/base -> origin/gh/soulitzer/387/base 2025-12-04T09:23:40.0094943Z * [new branch] gh/soulitzer/387/head -> origin/gh/soulitzer/387/head 2025-12-04T09:23:40.0095182Z * [new branch] gh/soulitzer/387/orig -> origin/gh/soulitzer/387/orig 2025-12-04T09:23:40.0095359Z * [new branch] gh/soulitzer/388/base -> origin/gh/soulitzer/388/base 2025-12-04T09:23:40.0095514Z * [new branch] gh/soulitzer/388/head -> origin/gh/soulitzer/388/head 2025-12-04T09:23:40.0095667Z * [new branch] gh/soulitzer/388/orig -> origin/gh/soulitzer/388/orig 2025-12-04T09:23:40.0098922Z * [new branch] gh/soulitzer/389/base -> origin/gh/soulitzer/389/base 2025-12-04T09:23:40.0099320Z * [new branch] gh/soulitzer/389/head -> origin/gh/soulitzer/389/head 2025-12-04T09:23:40.0099475Z * [new branch] gh/soulitzer/389/orig -> origin/gh/soulitzer/389/orig 2025-12-04T09:23:40.0099625Z * [new branch] gh/soulitzer/390/base -> origin/gh/soulitzer/390/base 2025-12-04T09:23:40.0099782Z * [new branch] gh/soulitzer/390/head -> origin/gh/soulitzer/390/head 2025-12-04T09:23:40.0099934Z * [new branch] gh/soulitzer/390/orig -> origin/gh/soulitzer/390/orig 2025-12-04T09:23:40.0104065Z * [new branch] gh/soulitzer/391/base -> origin/gh/soulitzer/391/base 2025-12-04T09:23:40.0110503Z * [new branch] gh/soulitzer/391/head -> origin/gh/soulitzer/391/head 2025-12-04T09:23:40.0113023Z * [new branch] gh/soulitzer/391/orig -> origin/gh/soulitzer/391/orig 2025-12-04T09:23:40.0113296Z * [new branch] gh/soulitzer/392/base -> origin/gh/soulitzer/392/base 2025-12-04T09:23:40.0113483Z * [new branch] gh/soulitzer/392/head -> origin/gh/soulitzer/392/head 2025-12-04T09:23:40.0113642Z * [new branch] gh/soulitzer/392/orig -> origin/gh/soulitzer/392/orig 2025-12-04T09:23:40.0113872Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-12-04T09:23:40.0118302Z * [new branch] gh/swolchok/819/base -> origin/gh/swolchok/819/base 2025-12-04T09:23:40.0118688Z * [new branch] gh/swolchok/819/head -> origin/gh/swolchok/819/head 2025-12-04T09:23:40.0118859Z * [new branch] gh/swolchok/819/orig -> origin/gh/swolchok/819/orig 2025-12-04T09:23:40.0119013Z * [new branch] gh/swolchok/824/base -> origin/gh/swolchok/824/base 2025-12-04T09:23:40.0119254Z * [new branch] gh/swolchok/824/head -> origin/gh/swolchok/824/head 2025-12-04T09:23:40.0119420Z * [new branch] gh/swolchok/824/orig -> origin/gh/swolchok/824/orig 2025-12-04T09:23:40.0119572Z * [new branch] gh/swolchok/829/base -> origin/gh/swolchok/829/base 2025-12-04T09:23:40.0119729Z * [new branch] gh/swolchok/829/head -> origin/gh/swolchok/829/head 2025-12-04T09:23:40.0119878Z * [new branch] gh/swolchok/829/orig -> origin/gh/swolchok/829/orig 2025-12-04T09:23:40.0120027Z * [new branch] gh/swolchok/839/base -> origin/gh/swolchok/839/base 2025-12-04T09:23:40.0120191Z * [new branch] gh/swolchok/839/head -> origin/gh/swolchok/839/head 2025-12-04T09:23:40.0120631Z * [new branch] gh/swolchok/839/orig -> origin/gh/swolchok/839/orig 2025-12-04T09:23:40.0120802Z * [new branch] gh/swolchok/841/base -> origin/gh/swolchok/841/base 2025-12-04T09:23:40.0120957Z * [new branch] gh/swolchok/841/head -> origin/gh/swolchok/841/head 2025-12-04T09:23:40.0121105Z * [new branch] gh/swolchok/841/orig -> origin/gh/swolchok/841/orig 2025-12-04T09:23:40.0121253Z * [new branch] gh/swolchok/842/base -> origin/gh/swolchok/842/base 2025-12-04T09:23:40.0121405Z * [new branch] gh/swolchok/842/head -> origin/gh/swolchok/842/head 2025-12-04T09:23:40.0121560Z * [new branch] gh/swolchok/842/orig -> origin/gh/swolchok/842/orig 2025-12-04T09:23:40.0121757Z * [new branch] gh/swolchok/845/base -> origin/gh/swolchok/845/base 2025-12-04T09:23:40.0121906Z * [new branch] gh/swolchok/845/head -> origin/gh/swolchok/845/head 2025-12-04T09:23:40.0122056Z * [new branch] gh/swolchok/845/orig -> origin/gh/swolchok/845/orig 2025-12-04T09:23:40.0126915Z * [new branch] gh/swolchok/848/base -> origin/gh/swolchok/848/base 2025-12-04T09:23:40.0132468Z * [new branch] gh/swolchok/848/head -> origin/gh/swolchok/848/head 2025-12-04T09:23:40.0138081Z * [new branch] gh/swolchok/848/orig -> origin/gh/swolchok/848/orig 2025-12-04T09:23:40.0141866Z * [new branch] gh/swolchok/856/base -> origin/gh/swolchok/856/base 2025-12-04T09:23:40.0146477Z * [new branch] gh/swolchok/856/head -> origin/gh/swolchok/856/head 2025-12-04T09:23:40.0151581Z * [new branch] gh/swolchok/856/orig -> origin/gh/swolchok/856/orig 2025-12-04T09:23:40.0153849Z * [new branch] gh/swolchok/860/base -> origin/gh/swolchok/860/base 2025-12-04T09:23:40.0154128Z * [new branch] gh/swolchok/860/head -> origin/gh/swolchok/860/head 2025-12-04T09:23:40.0159363Z * [new branch] gh/swolchok/860/orig -> origin/gh/swolchok/860/orig 2025-12-04T09:23:40.0159559Z * [new branch] gh/swolchok/861/base -> origin/gh/swolchok/861/base 2025-12-04T09:23:40.0159734Z * [new branch] gh/swolchok/861/head -> origin/gh/swolchok/861/head 2025-12-04T09:23:40.0159881Z * [new branch] gh/swolchok/861/orig -> origin/gh/swolchok/861/orig 2025-12-04T09:23:40.0160027Z * [new branch] gh/swolchok/862/base -> origin/gh/swolchok/862/base 2025-12-04T09:23:40.0160183Z * [new branch] gh/swolchok/862/head -> origin/gh/swolchok/862/head 2025-12-04T09:23:40.0160548Z * [new branch] gh/swolchok/862/orig -> origin/gh/swolchok/862/orig 2025-12-04T09:23:40.0160971Z * [new branch] gh/swolchok/863/base -> origin/gh/swolchok/863/base 2025-12-04T09:23:40.0161114Z * [new branch] gh/swolchok/863/head -> origin/gh/swolchok/863/head 2025-12-04T09:23:40.0161255Z * [new branch] gh/swolchok/863/orig -> origin/gh/swolchok/863/orig 2025-12-04T09:23:40.0161485Z * [new branch] gh/swolchok/864/base -> origin/gh/swolchok/864/base 2025-12-04T09:23:40.0161629Z * [new branch] gh/swolchok/864/head -> origin/gh/swolchok/864/head 2025-12-04T09:23:40.0161770Z * [new branch] gh/swolchok/864/orig -> origin/gh/swolchok/864/orig 2025-12-04T09:23:40.0161921Z * [new branch] gh/swolchok/865/base -> origin/gh/swolchok/865/base 2025-12-04T09:23:40.0162063Z * [new branch] gh/swolchok/865/head -> origin/gh/swolchok/865/head 2025-12-04T09:23:40.0162213Z * [new branch] gh/swolchok/865/orig -> origin/gh/swolchok/865/orig 2025-12-04T09:23:40.0162371Z * [new branch] gh/swolchok/866/base -> origin/gh/swolchok/866/base 2025-12-04T09:23:40.0162511Z * [new branch] gh/swolchok/866/head -> origin/gh/swolchok/866/head 2025-12-04T09:23:40.0162660Z * [new branch] gh/swolchok/866/orig -> origin/gh/swolchok/866/orig 2025-12-04T09:23:40.0162801Z * [new branch] gh/swolchok/867/base -> origin/gh/swolchok/867/base 2025-12-04T09:23:40.0162947Z * [new branch] gh/swolchok/867/head -> origin/gh/swolchok/867/head 2025-12-04T09:23:40.0163087Z * [new branch] gh/swolchok/867/orig -> origin/gh/swolchok/867/orig 2025-12-04T09:23:40.0163226Z * [new branch] gh/swolchok/868/base -> origin/gh/swolchok/868/base 2025-12-04T09:23:40.0163372Z * [new branch] gh/swolchok/868/head -> origin/gh/swolchok/868/head 2025-12-04T09:23:40.0163509Z * [new branch] gh/swolchok/868/orig -> origin/gh/swolchok/868/orig 2025-12-04T09:23:40.0163665Z * [new branch] gh/swolchok/869/base -> origin/gh/swolchok/869/base 2025-12-04T09:23:40.0163805Z * [new branch] gh/swolchok/869/head -> origin/gh/swolchok/869/head 2025-12-04T09:23:40.0163954Z * [new branch] gh/swolchok/869/orig -> origin/gh/swolchok/869/orig 2025-12-04T09:23:40.0164102Z * [new branch] gh/swolchok/870/base -> origin/gh/swolchok/870/base 2025-12-04T09:23:40.0164241Z * [new branch] gh/swolchok/870/head -> origin/gh/swolchok/870/head 2025-12-04T09:23:40.0164389Z * [new branch] gh/swolchok/870/orig -> origin/gh/swolchok/870/orig 2025-12-04T09:23:40.0164526Z * [new branch] gh/swolchok/871/base -> origin/gh/swolchok/871/base 2025-12-04T09:23:40.0164665Z * [new branch] gh/swolchok/871/head -> origin/gh/swolchok/871/head 2025-12-04T09:23:40.0164809Z * [new branch] gh/swolchok/871/orig -> origin/gh/swolchok/871/orig 2025-12-04T09:23:40.0164964Z * [new branch] gh/teja-rao/4/base -> origin/gh/teja-rao/4/base 2025-12-04T09:23:40.0165106Z * [new branch] gh/teja-rao/4/head -> origin/gh/teja-rao/4/head 2025-12-04T09:23:40.0165248Z * [new branch] gh/teja-rao/4/orig -> origin/gh/teja-rao/4/orig 2025-12-04T09:23:40.0165393Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-12-04T09:23:40.0165534Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-12-04T09:23:40.0165670Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-12-04T09:23:40.0165804Z * [new branch] gh/tianyu-l/3/base -> origin/gh/tianyu-l/3/base 2025-12-04T09:23:40.0165947Z * [new branch] gh/tianyu-l/3/orig -> origin/gh/tianyu-l/3/orig 2025-12-04T09:23:40.0166135Z * [new branch] gh/tianyu-l/4/base -> origin/gh/tianyu-l/4/base 2025-12-04T09:23:40.0166477Z * [new branch] gh/tianyu-l/4/head -> origin/gh/tianyu-l/4/head 2025-12-04T09:23:40.0166637Z * [new branch] gh/tianyu-l/4/orig -> origin/gh/tianyu-l/4/orig 2025-12-04T09:23:40.0166944Z * [new branch] gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base 2025-12-04T09:23:40.0172672Z * [new branch] gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head 2025-12-04T09:23:40.0172873Z * [new branch] gh/tugsbayasgalan/10/orig -> origin/gh/tugsbayasgalan/10/orig 2025-12-04T09:23:40.0173375Z * [new branch] gh/tugsbayasgalan/13/base -> origin/gh/tugsbayasgalan/13/base 2025-12-04T09:23:40.0173577Z * [new branch] gh/tugsbayasgalan/13/head -> origin/gh/tugsbayasgalan/13/head 2025-12-04T09:23:40.0173752Z * [new branch] gh/tugsbayasgalan/13/orig -> origin/gh/tugsbayasgalan/13/orig 2025-12-04T09:23:40.0173931Z * [new branch] gh/tugsbayasgalan/17/base -> origin/gh/tugsbayasgalan/17/base 2025-12-04T09:23:40.0174109Z * [new branch] gh/tugsbayasgalan/17/head -> origin/gh/tugsbayasgalan/17/head 2025-12-04T09:23:40.0174285Z * [new branch] gh/tugsbayasgalan/17/orig -> origin/gh/tugsbayasgalan/17/orig 2025-12-04T09:23:40.0174484Z * [new branch] gh/tugsbayasgalan/2/base -> origin/gh/tugsbayasgalan/2/base 2025-12-04T09:23:40.0174684Z * [new branch] gh/tugsbayasgalan/2/head -> origin/gh/tugsbayasgalan/2/head 2025-12-04T09:23:40.0176666Z * [new branch] gh/tugsbayasgalan/2/orig -> origin/gh/tugsbayasgalan/2/orig 2025-12-04T09:23:40.0177114Z * [new branch] gh/tugsbayasgalan/28/base -> origin/gh/tugsbayasgalan/28/base 2025-12-04T09:23:40.0177309Z * [new branch] gh/tugsbayasgalan/28/head -> origin/gh/tugsbayasgalan/28/head 2025-12-04T09:23:40.0179547Z * [new branch] gh/tugsbayasgalan/28/orig -> origin/gh/tugsbayasgalan/28/orig 2025-12-04T09:23:40.0179752Z * [new branch] gh/tugsbayasgalan/32/base -> origin/gh/tugsbayasgalan/32/base 2025-12-04T09:23:40.0184136Z * [new branch] gh/tugsbayasgalan/32/head -> origin/gh/tugsbayasgalan/32/head 2025-12-04T09:23:40.0184390Z * [new branch] gh/tugsbayasgalan/32/orig -> origin/gh/tugsbayasgalan/32/orig 2025-12-04T09:23:40.0184781Z * [new branch] gh/tugsbayasgalan/35/base -> origin/gh/tugsbayasgalan/35/base 2025-12-04T09:23:40.0184961Z * [new branch] gh/tugsbayasgalan/35/head -> origin/gh/tugsbayasgalan/35/head 2025-12-04T09:23:40.0185138Z * [new branch] gh/tugsbayasgalan/35/orig -> origin/gh/tugsbayasgalan/35/orig 2025-12-04T09:23:40.0185301Z * [new branch] gh/tugsbayasgalan/36/base -> origin/gh/tugsbayasgalan/36/base 2025-12-04T09:23:40.0185466Z * [new branch] gh/tugsbayasgalan/36/head -> origin/gh/tugsbayasgalan/36/head 2025-12-04T09:23:40.0185655Z * [new branch] gh/tugsbayasgalan/36/orig -> origin/gh/tugsbayasgalan/36/orig 2025-12-04T09:23:40.0187168Z * [new branch] gh/tugsbayasgalan/37/base -> origin/gh/tugsbayasgalan/37/base 2025-12-04T09:23:40.0187391Z * [new branch] gh/tugsbayasgalan/37/head -> origin/gh/tugsbayasgalan/37/head 2025-12-04T09:23:40.0188012Z * [new branch] gh/tugsbayasgalan/37/orig -> origin/gh/tugsbayasgalan/37/orig 2025-12-04T09:23:40.0192287Z * [new branch] gh/tugsbayasgalan/43/base -> origin/gh/tugsbayasgalan/43/base 2025-12-04T09:23:40.0192495Z * [new branch] gh/tugsbayasgalan/43/head -> origin/gh/tugsbayasgalan/43/head 2025-12-04T09:23:40.0192674Z * [new branch] gh/tugsbayasgalan/43/orig -> origin/gh/tugsbayasgalan/43/orig 2025-12-04T09:23:40.0192852Z * [new branch] gh/tugsbayasgalan/48/base -> origin/gh/tugsbayasgalan/48/base 2025-12-04T09:23:40.0193204Z * [new branch] gh/tugsbayasgalan/48/head -> origin/gh/tugsbayasgalan/48/head 2025-12-04T09:23:40.0193384Z * [new branch] gh/tugsbayasgalan/48/orig -> origin/gh/tugsbayasgalan/48/orig 2025-12-04T09:23:40.0193893Z * [new branch] gh/tugsbayasgalan/51/base -> origin/gh/tugsbayasgalan/51/base 2025-12-04T09:23:40.0194385Z * [new branch] gh/tugsbayasgalan/51/head -> origin/gh/tugsbayasgalan/51/head 2025-12-04T09:23:40.0194980Z * [new branch] gh/tugsbayasgalan/51/orig -> origin/gh/tugsbayasgalan/51/orig 2025-12-04T09:23:40.0196093Z * [new branch] gh/tugsbayasgalan/52/base -> origin/gh/tugsbayasgalan/52/base 2025-12-04T09:23:40.0196462Z * [new branch] gh/tugsbayasgalan/52/head -> origin/gh/tugsbayasgalan/52/head 2025-12-04T09:23:40.0197526Z * [new branch] gh/tugsbayasgalan/52/orig -> origin/gh/tugsbayasgalan/52/orig 2025-12-04T09:23:40.0198392Z * [new branch] gh/tugsbayasgalan/53/base -> origin/gh/tugsbayasgalan/53/base 2025-12-04T09:23:40.0198893Z * [new branch] gh/tugsbayasgalan/53/head -> origin/gh/tugsbayasgalan/53/head 2025-12-04T09:23:40.0199912Z * [new branch] gh/tugsbayasgalan/53/orig -> origin/gh/tugsbayasgalan/53/orig 2025-12-04T09:23:40.0201285Z * [new branch] gh/tugsbayasgalan/55/base -> origin/gh/tugsbayasgalan/55/base 2025-12-04T09:23:40.0201498Z * [new branch] gh/tugsbayasgalan/55/head -> origin/gh/tugsbayasgalan/55/head 2025-12-04T09:23:40.0202829Z * [new branch] gh/tugsbayasgalan/55/orig -> origin/gh/tugsbayasgalan/55/orig 2025-12-04T09:23:40.0203969Z * [new branch] gh/tugsbayasgalan/59/base -> origin/gh/tugsbayasgalan/59/base 2025-12-04T09:23:40.0204300Z * [new branch] gh/tugsbayasgalan/59/head -> origin/gh/tugsbayasgalan/59/head 2025-12-04T09:23:40.0205308Z * [new branch] gh/tugsbayasgalan/59/orig -> origin/gh/tugsbayasgalan/59/orig 2025-12-04T09:23:40.0206006Z * [new branch] gh/tugsbayasgalan/6/base -> origin/gh/tugsbayasgalan/6/base 2025-12-04T09:23:40.0207147Z * [new branch] gh/tugsbayasgalan/6/head -> origin/gh/tugsbayasgalan/6/head 2025-12-04T09:23:40.0207464Z * [new branch] gh/tugsbayasgalan/6/orig -> origin/gh/tugsbayasgalan/6/orig 2025-12-04T09:23:40.0208926Z * [new branch] gh/tugsbayasgalan/60/base -> origin/gh/tugsbayasgalan/60/base 2025-12-04T09:23:40.0209253Z * [new branch] gh/tugsbayasgalan/60/head -> origin/gh/tugsbayasgalan/60/head 2025-12-04T09:23:40.0210299Z * [new branch] gh/tugsbayasgalan/60/orig -> origin/gh/tugsbayasgalan/60/orig 2025-12-04T09:23:40.0211561Z * [new branch] gh/tugsbayasgalan/61/base -> origin/gh/tugsbayasgalan/61/base 2025-12-04T09:23:40.0211780Z * [new branch] gh/tugsbayasgalan/61/head -> origin/gh/tugsbayasgalan/61/head 2025-12-04T09:23:40.0212877Z * [new branch] gh/tugsbayasgalan/61/orig -> origin/gh/tugsbayasgalan/61/orig 2025-12-04T09:23:40.0213933Z * [new branch] gh/tugsbayasgalan/63/base -> origin/gh/tugsbayasgalan/63/base 2025-12-04T09:23:40.0214182Z * [new branch] gh/tugsbayasgalan/63/head -> origin/gh/tugsbayasgalan/63/head 2025-12-04T09:23:40.0215233Z * [new branch] gh/tugsbayasgalan/63/orig -> origin/gh/tugsbayasgalan/63/orig 2025-12-04T09:23:40.0215943Z * [new branch] gh/tugsbayasgalan/67/base -> origin/gh/tugsbayasgalan/67/base 2025-12-04T09:23:40.0216576Z * [new branch] gh/tugsbayasgalan/67/head -> origin/gh/tugsbayasgalan/67/head 2025-12-04T09:23:40.0217795Z * [new branch] gh/tugsbayasgalan/67/orig -> origin/gh/tugsbayasgalan/67/orig 2025-12-04T09:23:40.0218866Z * [new branch] gh/tugsbayasgalan/68/base -> origin/gh/tugsbayasgalan/68/base 2025-12-04T09:23:40.0219158Z * [new branch] gh/tugsbayasgalan/68/head -> origin/gh/tugsbayasgalan/68/head 2025-12-04T09:23:40.0220659Z * [new branch] gh/tugsbayasgalan/68/orig -> origin/gh/tugsbayasgalan/68/orig 2025-12-04T09:23:40.0220848Z * [new branch] gh/tugsbayasgalan/7/base -> origin/gh/tugsbayasgalan/7/base 2025-12-04T09:23:40.0224097Z * [new branch] gh/tugsbayasgalan/7/head -> origin/gh/tugsbayasgalan/7/head 2025-12-04T09:23:40.0224534Z * [new branch] gh/tugsbayasgalan/7/orig -> origin/gh/tugsbayasgalan/7/orig 2025-12-04T09:23:40.0224721Z * [new branch] gh/tugsbayasgalan/70/base -> origin/gh/tugsbayasgalan/70/base 2025-12-04T09:23:40.0224896Z * [new branch] gh/tugsbayasgalan/70/head -> origin/gh/tugsbayasgalan/70/head 2025-12-04T09:23:40.0225101Z * [new branch] gh/tugsbayasgalan/70/orig -> origin/gh/tugsbayasgalan/70/orig 2025-12-04T09:23:40.0230138Z * [new branch] gh/tugsbayasgalan/71/base -> origin/gh/tugsbayasgalan/71/base 2025-12-04T09:23:40.0230362Z * [new branch] gh/tugsbayasgalan/71/head -> origin/gh/tugsbayasgalan/71/head 2025-12-04T09:23:40.0230528Z * [new branch] gh/tugsbayasgalan/71/orig -> origin/gh/tugsbayasgalan/71/orig 2025-12-04T09:23:40.0230724Z * [new branch] gh/tugsbayasgalan/72/base -> origin/gh/tugsbayasgalan/72/base 2025-12-04T09:23:40.0230902Z * [new branch] gh/tugsbayasgalan/72/head -> origin/gh/tugsbayasgalan/72/head 2025-12-04T09:23:40.0231087Z * [new branch] gh/tugsbayasgalan/72/orig -> origin/gh/tugsbayasgalan/72/orig 2025-12-04T09:23:40.0231638Z * [new branch] gh/tugsbayasgalan/73/base -> origin/gh/tugsbayasgalan/73/base 2025-12-04T09:23:40.0235132Z * [new branch] gh/tugsbayasgalan/73/head -> origin/gh/tugsbayasgalan/73/head 2025-12-04T09:23:40.0235497Z * [new branch] gh/tugsbayasgalan/73/orig -> origin/gh/tugsbayasgalan/73/orig 2025-12-04T09:23:40.0235758Z * [new branch] gh/tugsbayasgalan/74/base -> origin/gh/tugsbayasgalan/74/base 2025-12-04T09:23:40.0236028Z * [new branch] gh/tugsbayasgalan/74/head -> origin/gh/tugsbayasgalan/74/head 2025-12-04T09:23:40.0236209Z * [new branch] gh/tugsbayasgalan/74/orig -> origin/gh/tugsbayasgalan/74/orig 2025-12-04T09:23:40.0237522Z * [new branch] gh/tugsbayasgalan/75/base -> origin/gh/tugsbayasgalan/75/base 2025-12-04T09:23:40.0238188Z * [new branch] gh/tugsbayasgalan/75/head -> origin/gh/tugsbayasgalan/75/head 2025-12-04T09:23:40.0238679Z * [new branch] gh/tugsbayasgalan/75/orig -> origin/gh/tugsbayasgalan/75/orig 2025-12-04T09:23:40.0239801Z * [new branch] gh/tugsbayasgalan/76/base -> origin/gh/tugsbayasgalan/76/base 2025-12-04T09:23:40.0240148Z * [new branch] gh/tugsbayasgalan/76/head -> origin/gh/tugsbayasgalan/76/head 2025-12-04T09:23:40.0240929Z * [new branch] gh/tugsbayasgalan/76/orig -> origin/gh/tugsbayasgalan/76/orig 2025-12-04T09:23:40.0243471Z * [new branch] gh/tugsbayasgalan/77/base -> origin/gh/tugsbayasgalan/77/base 2025-12-04T09:23:40.0243839Z * [new branch] gh/tugsbayasgalan/77/head -> origin/gh/tugsbayasgalan/77/head 2025-12-04T09:23:40.0244116Z * [new branch] gh/tugsbayasgalan/77/orig -> origin/gh/tugsbayasgalan/77/orig 2025-12-04T09:23:40.0244530Z * [new branch] gh/tugsbayasgalan/78/base -> origin/gh/tugsbayasgalan/78/base 2025-12-04T09:23:40.0245995Z * [new branch] gh/tugsbayasgalan/78/head -> origin/gh/tugsbayasgalan/78/head 2025-12-04T09:23:40.0249587Z * [new branch] gh/tugsbayasgalan/78/orig -> origin/gh/tugsbayasgalan/78/orig 2025-12-04T09:23:40.0255151Z * [new branch] gh/tugsbayasgalan/79/base -> origin/gh/tugsbayasgalan/79/base 2025-12-04T09:23:40.0260143Z * [new branch] gh/tugsbayasgalan/79/head -> origin/gh/tugsbayasgalan/79/head 2025-12-04T09:23:40.0264979Z * [new branch] gh/tugsbayasgalan/79/orig -> origin/gh/tugsbayasgalan/79/orig 2025-12-04T09:23:40.0269158Z * [new branch] gh/tugsbayasgalan/8/base -> origin/gh/tugsbayasgalan/8/base 2025-12-04T09:23:40.0271028Z * [new branch] gh/tugsbayasgalan/8/head -> origin/gh/tugsbayasgalan/8/head 2025-12-04T09:23:40.0271407Z * [new branch] gh/tugsbayasgalan/8/orig -> origin/gh/tugsbayasgalan/8/orig 2025-12-04T09:23:40.0271622Z * [new branch] gh/tugsbayasgalan/80/base -> origin/gh/tugsbayasgalan/80/base 2025-12-04T09:23:40.0271790Z * [new branch] gh/tugsbayasgalan/80/head -> origin/gh/tugsbayasgalan/80/head 2025-12-04T09:23:40.0271951Z * [new branch] gh/tugsbayasgalan/80/orig -> origin/gh/tugsbayasgalan/80/orig 2025-12-04T09:23:40.0272117Z * [new branch] gh/tugsbayasgalan/81/base -> origin/gh/tugsbayasgalan/81/base 2025-12-04T09:23:40.0272274Z * [new branch] gh/tugsbayasgalan/81/head -> origin/gh/tugsbayasgalan/81/head 2025-12-04T09:23:40.0272441Z * [new branch] gh/tugsbayasgalan/81/orig -> origin/gh/tugsbayasgalan/81/orig 2025-12-04T09:23:40.0272608Z * [new branch] gh/tugsbayasgalan/82/base -> origin/gh/tugsbayasgalan/82/base 2025-12-04T09:23:40.0272766Z * [new branch] gh/tugsbayasgalan/82/head -> origin/gh/tugsbayasgalan/82/head 2025-12-04T09:23:40.0272933Z * [new branch] gh/tugsbayasgalan/82/orig -> origin/gh/tugsbayasgalan/82/orig 2025-12-04T09:23:40.0273091Z * [new branch] gh/tugsbayasgalan/83/base -> origin/gh/tugsbayasgalan/83/base 2025-12-04T09:23:40.0273245Z * [new branch] gh/tugsbayasgalan/83/head -> origin/gh/tugsbayasgalan/83/head 2025-12-04T09:23:40.0273410Z * [new branch] gh/tugsbayasgalan/83/orig -> origin/gh/tugsbayasgalan/83/orig 2025-12-04T09:23:40.0273566Z * [new branch] gh/tugsbayasgalan/84/base -> origin/gh/tugsbayasgalan/84/base 2025-12-04T09:23:40.0273733Z * [new branch] gh/tugsbayasgalan/84/head -> origin/gh/tugsbayasgalan/84/head 2025-12-04T09:23:40.0273889Z * [new branch] gh/tugsbayasgalan/84/orig -> origin/gh/tugsbayasgalan/84/orig 2025-12-04T09:23:40.0274046Z * [new branch] gh/tugsbayasgalan/85/base -> origin/gh/tugsbayasgalan/85/base 2025-12-04T09:23:40.0274211Z * [new branch] gh/tugsbayasgalan/85/head -> origin/gh/tugsbayasgalan/85/head 2025-12-04T09:23:40.0274368Z * [new branch] gh/tugsbayasgalan/85/orig -> origin/gh/tugsbayasgalan/85/orig 2025-12-04T09:23:40.0274531Z * [new branch] gh/tugsbayasgalan/86/base -> origin/gh/tugsbayasgalan/86/base 2025-12-04T09:23:40.0274692Z * [new branch] gh/tugsbayasgalan/86/head -> origin/gh/tugsbayasgalan/86/head 2025-12-04T09:23:40.0274849Z * [new branch] gh/tugsbayasgalan/86/orig -> origin/gh/tugsbayasgalan/86/orig 2025-12-04T09:23:40.0275014Z * [new branch] gh/tugsbayasgalan/87/base -> origin/gh/tugsbayasgalan/87/base 2025-12-04T09:23:40.0275175Z * [new branch] gh/tugsbayasgalan/87/head -> origin/gh/tugsbayasgalan/87/head 2025-12-04T09:23:40.0275336Z * [new branch] gh/tugsbayasgalan/87/orig -> origin/gh/tugsbayasgalan/87/orig 2025-12-04T09:23:40.0275780Z * [new branch] gh/tugsbayasgalan/88/base -> origin/gh/tugsbayasgalan/88/base 2025-12-04T09:23:40.0275944Z * [new branch] gh/tugsbayasgalan/88/head -> origin/gh/tugsbayasgalan/88/head 2025-12-04T09:23:40.0276208Z * [new branch] gh/tugsbayasgalan/88/orig -> origin/gh/tugsbayasgalan/88/orig 2025-12-04T09:23:40.0276507Z * [new branch] gh/tugsbayasgalan/89/base -> origin/gh/tugsbayasgalan/89/base 2025-12-04T09:23:40.0276683Z * [new branch] gh/tugsbayasgalan/89/head -> origin/gh/tugsbayasgalan/89/head 2025-12-04T09:23:40.0276849Z * [new branch] gh/tugsbayasgalan/89/orig -> origin/gh/tugsbayasgalan/89/orig 2025-12-04T09:23:40.0281565Z * [new branch] gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base 2025-12-04T09:23:40.0287573Z * [new branch] gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head 2025-12-04T09:23:40.0292819Z * [new branch] gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig 2025-12-04T09:23:40.0297676Z * [new branch] gh/tugsbayasgalan/90/base -> origin/gh/tugsbayasgalan/90/base 2025-12-04T09:23:40.0303166Z * [new branch] gh/tugsbayasgalan/90/head -> origin/gh/tugsbayasgalan/90/head 2025-12-04T09:23:40.0307953Z * [new branch] gh/tugsbayasgalan/90/orig -> origin/gh/tugsbayasgalan/90/orig 2025-12-04T09:23:40.0310274Z * [new branch] gh/tugsbayasgalan/91/base -> origin/gh/tugsbayasgalan/91/base 2025-12-04T09:23:40.0310479Z * [new branch] gh/tugsbayasgalan/91/head -> origin/gh/tugsbayasgalan/91/head 2025-12-04T09:23:40.0310666Z * [new branch] gh/tugsbayasgalan/91/orig -> origin/gh/tugsbayasgalan/91/orig 2025-12-04T09:23:40.0310822Z * [new branch] gh/tugsbayasgalan/92/base -> origin/gh/tugsbayasgalan/92/base 2025-12-04T09:23:40.0310984Z * [new branch] gh/tugsbayasgalan/92/head -> origin/gh/tugsbayasgalan/92/head 2025-12-04T09:23:40.0311143Z * [new branch] gh/tugsbayasgalan/92/orig -> origin/gh/tugsbayasgalan/92/orig 2025-12-04T09:23:40.0311298Z * [new branch] gh/tugsbayasgalan/93/base -> origin/gh/tugsbayasgalan/93/base 2025-12-04T09:23:40.0311458Z * [new branch] gh/tugsbayasgalan/93/head -> origin/gh/tugsbayasgalan/93/head 2025-12-04T09:23:40.0311608Z * [new branch] gh/tugsbayasgalan/93/orig -> origin/gh/tugsbayasgalan/93/orig 2025-12-04T09:23:40.0311750Z * [new branch] gh/v0i0/14/base -> origin/gh/v0i0/14/base 2025-12-04T09:23:40.0311879Z * [new branch] gh/v0i0/14/head -> origin/gh/v0i0/14/head 2025-12-04T09:23:40.0312004Z * [new branch] gh/v0i0/14/orig -> origin/gh/v0i0/14/orig 2025-12-04T09:23:40.0312139Z * [new branch] gh/v0i0/15/base -> origin/gh/v0i0/15/base 2025-12-04T09:23:40.0312257Z * [new branch] gh/v0i0/15/head -> origin/gh/v0i0/15/head 2025-12-04T09:23:40.0312379Z * [new branch] gh/v0i0/15/orig -> origin/gh/v0i0/15/orig 2025-12-04T09:23:40.0312505Z * [new branch] gh/v0i0/16/base -> origin/gh/v0i0/16/base 2025-12-04T09:23:40.0312625Z * [new branch] gh/v0i0/16/head -> origin/gh/v0i0/16/head 2025-12-04T09:23:40.0312750Z * [new branch] gh/v0i0/16/orig -> origin/gh/v0i0/16/orig 2025-12-04T09:23:40.0312869Z * [new branch] gh/v0i0/17/base -> origin/gh/v0i0/17/base 2025-12-04T09:23:40.0312988Z * [new branch] gh/v0i0/17/head -> origin/gh/v0i0/17/head 2025-12-04T09:23:40.0313118Z * [new branch] gh/v0i0/17/orig -> origin/gh/v0i0/17/orig 2025-12-04T09:23:40.0313247Z * [new branch] gh/v0i0/18/base -> origin/gh/v0i0/18/base 2025-12-04T09:23:40.0313372Z * [new branch] gh/v0i0/18/head -> origin/gh/v0i0/18/head 2025-12-04T09:23:40.0313493Z * [new branch] gh/v0i0/18/orig -> origin/gh/v0i0/18/orig 2025-12-04T09:23:40.0313612Z * [new branch] gh/v0i0/19/base -> origin/gh/v0i0/19/base 2025-12-04T09:23:40.0313739Z * [new branch] gh/v0i0/19/head -> origin/gh/v0i0/19/head 2025-12-04T09:23:40.0313858Z * [new branch] gh/v0i0/19/orig -> origin/gh/v0i0/19/orig 2025-12-04T09:23:40.0314029Z * [new branch] gh/vishal9-team/1/base -> origin/gh/vishal9-team/1/base 2025-12-04T09:23:40.0314180Z * [new branch] gh/vishal9-team/1/head -> origin/gh/vishal9-team/1/head 2025-12-04T09:23:40.0314496Z * [new branch] gh/vishal9-team/2/base -> origin/gh/vishal9-team/2/base 2025-12-04T09:23:40.0314645Z * [new branch] gh/vishal9-team/2/head -> origin/gh/vishal9-team/2/head 2025-12-04T09:23:40.0314784Z * [new branch] gh/vishal9-team/2/orig -> origin/gh/vishal9-team/2/orig 2025-12-04T09:23:40.0314979Z * [new branch] gh/vishal9-team/3/base -> origin/gh/vishal9-team/3/base 2025-12-04T09:23:40.0315123Z * [new branch] gh/vishal9-team/3/head -> origin/gh/vishal9-team/3/head 2025-12-04T09:23:40.0315265Z * [new branch] gh/vishal9-team/3/orig -> origin/gh/vishal9-team/3/orig 2025-12-04T09:23:40.0315603Z * [new branch] gh/vishal9-team/4/base -> origin/gh/vishal9-team/4/base 2025-12-04T09:23:40.0315777Z * [new branch] gh/vishal9-team/4/head -> origin/gh/vishal9-team/4/head 2025-12-04T09:23:40.0316027Z * [new branch] gh/vishal9-team/4/orig -> origin/gh/vishal9-team/4/orig 2025-12-04T09:23:40.0316397Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-12-04T09:23:40.0316554Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-12-04T09:23:40.0316696Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-12-04T09:23:40.0316951Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-12-04T09:23:40.0318737Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-12-04T09:23:40.0318922Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-12-04T09:23:40.0324130Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-12-04T09:23:40.0324453Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-12-04T09:23:40.0324631Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-12-04T09:23:40.0324882Z * [new branch] gh/wconstab/444/base -> origin/gh/wconstab/444/base 2025-12-04T09:23:40.0329199Z * [new branch] gh/wconstab/444/head -> origin/gh/wconstab/444/head 2025-12-04T09:23:40.0329537Z * [new branch] gh/wconstab/444/orig -> origin/gh/wconstab/444/orig 2025-12-04T09:23:40.0329735Z * [new branch] gh/wconstab/447/base -> origin/gh/wconstab/447/base 2025-12-04T09:23:40.0329879Z * [new branch] gh/wconstab/447/head -> origin/gh/wconstab/447/head 2025-12-04T09:23:40.0330023Z * [new branch] gh/wconstab/447/orig -> origin/gh/wconstab/447/orig 2025-12-04T09:23:40.0330290Z * [new branch] gh/wconstab/448/base -> origin/gh/wconstab/448/base 2025-12-04T09:23:40.0330699Z * [new branch] gh/wconstab/448/head -> origin/gh/wconstab/448/head 2025-12-04T09:23:40.0331198Z * [new branch] gh/wconstab/448/orig -> origin/gh/wconstab/448/orig 2025-12-04T09:23:40.0331386Z * [new branch] gh/wconstab/449/base -> origin/gh/wconstab/449/base 2025-12-04T09:23:40.0331539Z * [new branch] gh/wconstab/449/head -> origin/gh/wconstab/449/head 2025-12-04T09:23:40.0331711Z * [new branch] gh/wconstab/449/orig -> origin/gh/wconstab/449/orig 2025-12-04T09:23:40.0331873Z * [new branch] gh/wconstab/450/base -> origin/gh/wconstab/450/base 2025-12-04T09:23:40.0332502Z * [new branch] gh/wconstab/450/head -> origin/gh/wconstab/450/head 2025-12-04T09:23:40.0333068Z * [new branch] gh/wconstab/450/orig -> origin/gh/wconstab/450/orig 2025-12-04T09:23:40.0334242Z * [new branch] gh/wconstab/451/base -> origin/gh/wconstab/451/base 2025-12-04T09:23:40.0334636Z * [new branch] gh/wconstab/451/head -> origin/gh/wconstab/451/head 2025-12-04T09:23:40.0337799Z * [new branch] gh/wconstab/451/orig -> origin/gh/wconstab/451/orig 2025-12-04T09:23:40.0337988Z * [new branch] gh/wconstab/452/base -> origin/gh/wconstab/452/base 2025-12-04T09:23:40.0338150Z * [new branch] gh/wconstab/452/head -> origin/gh/wconstab/452/head 2025-12-04T09:23:40.0338452Z * [new branch] gh/wconstab/452/orig -> origin/gh/wconstab/452/orig 2025-12-04T09:23:40.0338639Z * [new branch] gh/wconstab/453/base -> origin/gh/wconstab/453/base 2025-12-04T09:23:40.0339737Z * [new branch] gh/wconstab/453/head -> origin/gh/wconstab/453/head 2025-12-04T09:23:40.0340050Z * [new branch] gh/wconstab/453/orig -> origin/gh/wconstab/453/orig 2025-12-04T09:23:40.0341286Z * [new branch] gh/wconstab/454/base -> origin/gh/wconstab/454/base 2025-12-04T09:23:40.0341495Z * [new branch] gh/wconstab/454/head -> origin/gh/wconstab/454/head 2025-12-04T09:23:40.0343568Z * [new branch] gh/wconstab/454/orig -> origin/gh/wconstab/454/orig 2025-12-04T09:23:40.0344123Z * [new branch] gh/wconstab/455/base -> origin/gh/wconstab/455/base 2025-12-04T09:23:40.0344495Z * [new branch] gh/wconstab/455/head -> origin/gh/wconstab/455/head 2025-12-04T09:23:40.0344675Z * [new branch] gh/wconstab/455/orig -> origin/gh/wconstab/455/orig 2025-12-04T09:23:40.0346307Z * [new branch] gh/wconstab/456/base -> origin/gh/wconstab/456/base 2025-12-04T09:23:40.0347193Z * [new branch] gh/wconstab/456/head -> origin/gh/wconstab/456/head 2025-12-04T09:23:40.0348215Z * [new branch] gh/wconstab/456/orig -> origin/gh/wconstab/456/orig 2025-12-04T09:23:40.0349283Z * [new branch] gh/wconstab/457/base -> origin/gh/wconstab/457/base 2025-12-04T09:23:40.0349582Z * [new branch] gh/wconstab/457/head -> origin/gh/wconstab/457/head 2025-12-04T09:23:40.0350501Z * [new branch] gh/wconstab/457/orig -> origin/gh/wconstab/457/orig 2025-12-04T09:23:40.0353942Z * [new branch] gh/wconstab/458/base -> origin/gh/wconstab/458/base 2025-12-04T09:23:40.0354357Z * [new branch] gh/wconstab/458/head -> origin/gh/wconstab/458/head 2025-12-04T09:23:40.0354544Z * [new branch] gh/wconstab/458/orig -> origin/gh/wconstab/458/orig 2025-12-04T09:23:40.0354703Z * [new branch] gh/wconstab/459/base -> origin/gh/wconstab/459/base 2025-12-04T09:23:40.0354869Z * [new branch] gh/wconstab/459/head -> origin/gh/wconstab/459/head 2025-12-04T09:23:40.0355293Z * [new branch] gh/wconstab/459/orig -> origin/gh/wconstab/459/orig 2025-12-04T09:23:40.0359694Z * [new branch] gh/wconstab/460/base -> origin/gh/wconstab/460/base 2025-12-04T09:23:40.0359915Z * [new branch] gh/wconstab/460/head -> origin/gh/wconstab/460/head 2025-12-04T09:23:40.0360073Z * [new branch] gh/wconstab/460/orig -> origin/gh/wconstab/460/orig 2025-12-04T09:23:40.0360232Z * [new branch] gh/wconstab/461/base -> origin/gh/wconstab/461/base 2025-12-04T09:23:40.0360859Z * [new branch] gh/wconstab/461/head -> origin/gh/wconstab/461/head 2025-12-04T09:23:40.0361453Z * [new branch] gh/wconstab/461/orig -> origin/gh/wconstab/461/orig 2025-12-04T09:23:40.0363276Z * [new branch] gh/wconstab/462/base -> origin/gh/wconstab/462/base 2025-12-04T09:23:40.0363454Z * [new branch] gh/wconstab/462/head -> origin/gh/wconstab/462/head 2025-12-04T09:23:40.0363811Z * [new branch] gh/wconstab/462/orig -> origin/gh/wconstab/462/orig 2025-12-04T09:23:40.0366657Z * [new branch] gh/wconstab/463/base -> origin/gh/wconstab/463/base 2025-12-04T09:23:40.0367276Z * [new branch] gh/wconstab/463/head -> origin/gh/wconstab/463/head 2025-12-04T09:23:40.0367460Z * [new branch] gh/wconstab/463/orig -> origin/gh/wconstab/463/orig 2025-12-04T09:23:40.0367715Z * [new branch] gh/wconstab/464/base -> origin/gh/wconstab/464/base 2025-12-04T09:23:40.0368140Z * [new branch] gh/wconstab/464/head -> origin/gh/wconstab/464/head 2025-12-04T09:23:40.0368970Z * [new branch] gh/wconstab/464/orig -> origin/gh/wconstab/464/orig 2025-12-04T09:23:40.0372149Z * [new branch] gh/wconstab/465/base -> origin/gh/wconstab/465/base 2025-12-04T09:23:40.0372487Z * [new branch] gh/wconstab/465/head -> origin/gh/wconstab/465/head 2025-12-04T09:23:40.0372696Z * [new branch] gh/wconstab/465/orig -> origin/gh/wconstab/465/orig 2025-12-04T09:23:40.0372879Z * [new branch] gh/wconstab/466/base -> origin/gh/wconstab/466/base 2025-12-04T09:23:40.0373058Z * [new branch] gh/wconstab/466/head -> origin/gh/wconstab/466/head 2025-12-04T09:23:40.0373296Z * [new branch] gh/wconstab/466/orig -> origin/gh/wconstab/466/orig 2025-12-04T09:23:40.0378887Z * [new branch] gh/wconstab/467/base -> origin/gh/wconstab/467/base 2025-12-04T09:23:40.0383722Z * [new branch] gh/wconstab/467/head -> origin/gh/wconstab/467/head 2025-12-04T09:23:40.0385106Z * [new branch] gh/wconstab/467/orig -> origin/gh/wconstab/467/orig 2025-12-04T09:23:40.0385650Z * [new branch] gh/wconstab/468/base -> origin/gh/wconstab/468/base 2025-12-04T09:23:40.0385833Z * [new branch] gh/wconstab/468/head -> origin/gh/wconstab/468/head 2025-12-04T09:23:40.0386001Z * [new branch] gh/wconstab/468/orig -> origin/gh/wconstab/468/orig 2025-12-04T09:23:40.0386146Z * [new branch] gh/weifengpy/39/base -> origin/gh/weifengpy/39/base 2025-12-04T09:23:40.0386316Z * [new branch] gh/weifengpy/39/head -> origin/gh/weifengpy/39/head 2025-12-04T09:23:40.0386453Z * [new branch] gh/weifengpy/39/orig -> origin/gh/weifengpy/39/orig 2025-12-04T09:23:40.0386588Z * [new branch] gh/weifengpy/40/base -> origin/gh/weifengpy/40/base 2025-12-04T09:23:40.0386735Z * [new branch] gh/weifengpy/40/head -> origin/gh/weifengpy/40/head 2025-12-04T09:23:40.0386868Z * [new branch] gh/weifengpy/40/orig -> origin/gh/weifengpy/40/orig 2025-12-04T09:23:40.0387009Z * [new branch] gh/weifengpy/41/base -> origin/gh/weifengpy/41/base 2025-12-04T09:23:40.0387147Z * [new branch] gh/weifengpy/41/head -> origin/gh/weifengpy/41/head 2025-12-04T09:23:40.0387280Z * [new branch] gh/weifengpy/41/orig -> origin/gh/weifengpy/41/orig 2025-12-04T09:23:40.0389349Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-12-04T09:23:40.0390104Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-12-04T09:23:40.0390489Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-12-04T09:23:40.0390808Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-12-04T09:23:40.0391844Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-12-04T09:23:40.0392157Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-12-04T09:23:40.0394746Z * [new branch] gh/williamwen42/282/base -> origin/gh/williamwen42/282/base 2025-12-04T09:23:40.0394945Z * [new branch] gh/williamwen42/282/head -> origin/gh/williamwen42/282/head 2025-12-04T09:23:40.0395109Z * [new branch] gh/williamwen42/282/orig -> origin/gh/williamwen42/282/orig 2025-12-04T09:23:40.0395479Z * [new branch] gh/williamwen42/287/base -> origin/gh/williamwen42/287/base 2025-12-04T09:23:40.0396945Z * [new branch] gh/williamwen42/287/head -> origin/gh/williamwen42/287/head 2025-12-04T09:23:40.0397177Z * [new branch] gh/williamwen42/287/orig -> origin/gh/williamwen42/287/orig 2025-12-04T09:23:40.0397726Z * [new branch] gh/williamwen42/288/base -> origin/gh/williamwen42/288/base 2025-12-04T09:23:40.0398230Z * [new branch] gh/williamwen42/288/head -> origin/gh/williamwen42/288/head 2025-12-04T09:23:40.0399193Z * [new branch] gh/williamwen42/288/orig -> origin/gh/williamwen42/288/orig 2025-12-04T09:23:40.0400289Z * [new branch] gh/williamwen42/296/base -> origin/gh/williamwen42/296/base 2025-12-04T09:23:40.0400993Z * [new branch] gh/williamwen42/296/head -> origin/gh/williamwen42/296/head 2025-12-04T09:23:40.0402272Z * [new branch] gh/williamwen42/296/orig -> origin/gh/williamwen42/296/orig 2025-12-04T09:23:40.0402444Z * [new branch] gh/williamwen42/297/base -> origin/gh/williamwen42/297/base 2025-12-04T09:23:40.0403506Z * [new branch] gh/williamwen42/297/head -> origin/gh/williamwen42/297/head 2025-12-04T09:23:40.0403961Z * [new branch] gh/williamwen42/297/orig -> origin/gh/williamwen42/297/orig 2025-12-04T09:23:40.0405593Z * [new branch] gh/williamwen42/306/base -> origin/gh/williamwen42/306/base 2025-12-04T09:23:40.0405850Z * [new branch] gh/williamwen42/306/head -> origin/gh/williamwen42/306/head 2025-12-04T09:23:40.0406882Z * [new branch] gh/williamwen42/306/orig -> origin/gh/williamwen42/306/orig 2025-12-04T09:23:40.0407978Z * [new branch] gh/williamwen42/309/base -> origin/gh/williamwen42/309/base 2025-12-04T09:23:40.0408233Z * [new branch] gh/williamwen42/309/head -> origin/gh/williamwen42/309/head 2025-12-04T09:23:40.0409381Z * [new branch] gh/williamwen42/309/orig -> origin/gh/williamwen42/309/orig 2025-12-04T09:23:40.0409939Z * [new branch] gh/williamwen42/310/base -> origin/gh/williamwen42/310/base 2025-12-04T09:23:40.0410964Z * [new branch] gh/williamwen42/310/head -> origin/gh/williamwen42/310/head 2025-12-04T09:23:40.0411549Z * [new branch] gh/williamwen42/310/orig -> origin/gh/williamwen42/310/orig 2025-12-04T09:23:40.0413493Z * [new branch] gh/williamwen42/311/base -> origin/gh/williamwen42/311/base 2025-12-04T09:23:40.0413785Z * [new branch] gh/williamwen42/311/head -> origin/gh/williamwen42/311/head 2025-12-04T09:23:40.0414875Z * [new branch] gh/williamwen42/311/orig -> origin/gh/williamwen42/311/orig 2025-12-04T09:23:40.0415428Z * [new branch] gh/williamwen42/319/base -> origin/gh/williamwen42/319/base 2025-12-04T09:23:40.0416610Z * [new branch] gh/williamwen42/319/head -> origin/gh/williamwen42/319/head 2025-12-04T09:23:40.0416952Z * [new branch] gh/williamwen42/319/orig -> origin/gh/williamwen42/319/orig 2025-12-04T09:23:40.0418215Z * [new branch] gh/williamwen42/325/base -> origin/gh/williamwen42/325/base 2025-12-04T09:23:40.0419435Z * [new branch] gh/williamwen42/325/head -> origin/gh/williamwen42/325/head 2025-12-04T09:23:40.0419607Z * [new branch] gh/williamwen42/325/orig -> origin/gh/williamwen42/325/orig 2025-12-04T09:23:40.0420338Z * [new branch] gh/williamwen42/326/base -> origin/gh/williamwen42/326/base 2025-12-04T09:23:40.0423578Z * [new branch] gh/williamwen42/326/head -> origin/gh/williamwen42/326/head 2025-12-04T09:23:40.0423795Z * [new branch] gh/williamwen42/326/orig -> origin/gh/williamwen42/326/orig 2025-12-04T09:23:40.0423989Z * [new branch] gh/williamwen42/327/base -> origin/gh/williamwen42/327/base 2025-12-04T09:23:40.0424346Z * [new branch] gh/williamwen42/327/head -> origin/gh/williamwen42/327/head 2025-12-04T09:23:40.0424525Z * [new branch] gh/williamwen42/327/orig -> origin/gh/williamwen42/327/orig 2025-12-04T09:23:40.0424920Z * [new branch] gh/williamwen42/328/base -> origin/gh/williamwen42/328/base 2025-12-04T09:23:40.0426751Z * [new branch] gh/williamwen42/328/head -> origin/gh/williamwen42/328/head 2025-12-04T09:23:40.0426946Z * [new branch] gh/williamwen42/328/orig -> origin/gh/williamwen42/328/orig 2025-12-04T09:23:40.0427680Z * [new branch] gh/williamwen42/329/base -> origin/gh/williamwen42/329/base 2025-12-04T09:23:40.0428470Z * [new branch] gh/williamwen42/329/head -> origin/gh/williamwen42/329/head 2025-12-04T09:23:40.0429065Z * [new branch] gh/williamwen42/329/orig -> origin/gh/williamwen42/329/orig 2025-12-04T09:23:40.0433311Z * [new branch] gh/williamwen42/330/base -> origin/gh/williamwen42/330/base 2025-12-04T09:23:40.0433500Z * [new branch] gh/williamwen42/330/head -> origin/gh/williamwen42/330/head 2025-12-04T09:23:40.0433686Z * [new branch] gh/williamwen42/330/orig -> origin/gh/williamwen42/330/orig 2025-12-04T09:23:40.0433849Z * [new branch] gh/williamwen42/331/base -> origin/gh/williamwen42/331/base 2025-12-04T09:23:40.0434010Z * [new branch] gh/williamwen42/331/head -> origin/gh/williamwen42/331/head 2025-12-04T09:23:40.0434198Z * [new branch] gh/williamwen42/331/orig -> origin/gh/williamwen42/331/orig 2025-12-04T09:23:40.0434420Z * [new branch] gh/williamwen42/332/base -> origin/gh/williamwen42/332/base 2025-12-04T09:23:40.0437542Z * [new branch] gh/williamwen42/332/head -> origin/gh/williamwen42/332/head 2025-12-04T09:23:40.0437727Z * [new branch] gh/williamwen42/332/orig -> origin/gh/williamwen42/332/orig 2025-12-04T09:23:40.0437905Z * [new branch] gh/williamwen42/333/base -> origin/gh/williamwen42/333/base 2025-12-04T09:23:40.0438054Z * [new branch] gh/williamwen42/333/head -> origin/gh/williamwen42/333/head 2025-12-04T09:23:40.0438978Z * [new branch] gh/williamwen42/333/orig -> origin/gh/williamwen42/333/orig 2025-12-04T09:23:40.0439801Z * [new branch] gh/williamwen42/334/base -> origin/gh/williamwen42/334/base 2025-12-04T09:23:40.0441686Z * [new branch] gh/williamwen42/334/head -> origin/gh/williamwen42/334/head 2025-12-04T09:23:40.0441853Z * [new branch] gh/williamwen42/334/orig -> origin/gh/williamwen42/334/orig 2025-12-04T09:23:40.0445987Z * [new branch] gh/williamwen42/335/base -> origin/gh/williamwen42/335/base 2025-12-04T09:23:40.0446882Z * [new branch] gh/williamwen42/335/head -> origin/gh/williamwen42/335/head 2025-12-04T09:23:40.0447453Z * [new branch] gh/williamwen42/335/orig -> origin/gh/williamwen42/335/orig 2025-12-04T09:23:40.0451208Z * [new branch] gh/williamwen42/336/base -> origin/gh/williamwen42/336/base 2025-12-04T09:23:40.0451503Z * [new branch] gh/williamwen42/336/head -> origin/gh/williamwen42/336/head 2025-12-04T09:23:40.0452532Z * [new branch] gh/williamwen42/336/orig -> origin/gh/williamwen42/336/orig 2025-12-04T09:23:40.0456280Z * [new branch] gh/williamwen42/337/base -> origin/gh/williamwen42/337/base 2025-12-04T09:23:40.0456782Z * [new branch] gh/williamwen42/337/head -> origin/gh/williamwen42/337/head 2025-12-04T09:23:40.0456957Z * [new branch] gh/williamwen42/337/orig -> origin/gh/williamwen42/337/orig 2025-12-04T09:23:40.0457106Z * [new branch] gh/williamwen42/338/base -> origin/gh/williamwen42/338/base 2025-12-04T09:23:40.0457265Z * [new branch] gh/williamwen42/338/head -> origin/gh/williamwen42/338/head 2025-12-04T09:23:40.0463641Z * [new branch] gh/williamwen42/338/orig -> origin/gh/williamwen42/338/orig 2025-12-04T09:23:40.0464018Z * [new branch] gh/williamwen42/339/base -> origin/gh/williamwen42/339/base 2025-12-04T09:23:40.0464560Z * [new branch] gh/williamwen42/339/head -> origin/gh/williamwen42/339/head 2025-12-04T09:23:40.0464871Z * [new branch] gh/williamwen42/339/orig -> origin/gh/williamwen42/339/orig 2025-12-04T09:23:40.0465129Z * [new branch] gh/williamwen42/340/base -> origin/gh/williamwen42/340/base 2025-12-04T09:23:40.0465393Z * [new branch] gh/williamwen42/340/head -> origin/gh/williamwen42/340/head 2025-12-04T09:23:40.0466158Z * [new branch] gh/williamwen42/340/orig -> origin/gh/williamwen42/340/orig 2025-12-04T09:23:40.0466379Z * [new branch] gh/williamwen42/341/base -> origin/gh/williamwen42/341/base 2025-12-04T09:23:40.0466576Z * [new branch] gh/williamwen42/341/head -> origin/gh/williamwen42/341/head 2025-12-04T09:23:40.0466753Z * [new branch] gh/williamwen42/341/orig -> origin/gh/williamwen42/341/orig 2025-12-04T09:23:40.0466921Z * [new branch] gh/williamwen42/342/base -> origin/gh/williamwen42/342/base 2025-12-04T09:23:40.0467116Z * [new branch] gh/williamwen42/342/head -> origin/gh/williamwen42/342/head 2025-12-04T09:23:40.0467280Z * [new branch] gh/williamwen42/342/orig -> origin/gh/williamwen42/342/orig 2025-12-04T09:23:40.0467440Z * [new branch] gh/williamwen42/343/base -> origin/gh/williamwen42/343/base 2025-12-04T09:23:40.0472864Z * [new branch] gh/williamwen42/343/head -> origin/gh/williamwen42/343/head 2025-12-04T09:23:40.0473100Z * [new branch] gh/williamwen42/343/orig -> origin/gh/williamwen42/343/orig 2025-12-04T09:23:40.0473289Z * [new branch] gh/williamwen42/344/base -> origin/gh/williamwen42/344/base 2025-12-04T09:23:40.0473452Z * [new branch] gh/williamwen42/344/head -> origin/gh/williamwen42/344/head 2025-12-04T09:23:40.0473615Z * [new branch] gh/williamwen42/344/orig -> origin/gh/williamwen42/344/orig 2025-12-04T09:23:40.0473789Z * [new branch] gh/williamwen42/345/base -> origin/gh/williamwen42/345/base 2025-12-04T09:23:40.0473956Z * [new branch] gh/williamwen42/345/head -> origin/gh/williamwen42/345/head 2025-12-04T09:23:40.0479294Z * [new branch] gh/williamwen42/345/orig -> origin/gh/williamwen42/345/orig 2025-12-04T09:23:40.0479513Z * [new branch] gh/williamwen42/346/base -> origin/gh/williamwen42/346/base 2025-12-04T09:23:40.0479680Z * [new branch] gh/williamwen42/346/head -> origin/gh/williamwen42/346/head 2025-12-04T09:23:40.0479854Z * [new branch] gh/williamwen42/346/orig -> origin/gh/williamwen42/346/orig 2025-12-04T09:23:40.0480038Z * [new branch] gh/williamwen42/347/base -> origin/gh/williamwen42/347/base 2025-12-04T09:23:40.0480206Z * [new branch] gh/williamwen42/347/head -> origin/gh/williamwen42/347/head 2025-12-04T09:23:40.0480656Z * [new branch] gh/williamwen42/347/orig -> origin/gh/williamwen42/347/orig 2025-12-04T09:23:40.0480854Z * [new branch] gh/williamwen42/348/base -> origin/gh/williamwen42/348/base 2025-12-04T09:23:40.0481018Z * [new branch] gh/williamwen42/348/head -> origin/gh/williamwen42/348/head 2025-12-04T09:23:40.0481184Z * [new branch] gh/williamwen42/348/orig -> origin/gh/williamwen42/348/orig 2025-12-04T09:23:40.0481355Z * [new branch] gh/williamwen42/349/base -> origin/gh/williamwen42/349/base 2025-12-04T09:23:40.0481518Z * [new branch] gh/williamwen42/349/head -> origin/gh/williamwen42/349/head 2025-12-04T09:23:40.0481811Z * [new branch] gh/williamwen42/349/orig -> origin/gh/williamwen42/349/orig 2025-12-04T09:23:40.0481963Z * [new branch] gh/williamwen42/350/base -> origin/gh/williamwen42/350/base 2025-12-04T09:23:40.0488123Z * [new branch] gh/williamwen42/350/head -> origin/gh/williamwen42/350/head 2025-12-04T09:23:40.0491934Z * [new branch] gh/williamwen42/350/orig -> origin/gh/williamwen42/350/orig 2025-12-04T09:23:40.0495025Z * [new branch] gh/williamwen42/351/base -> origin/gh/williamwen42/351/base 2025-12-04T09:23:40.0495195Z * [new branch] gh/williamwen42/351/head -> origin/gh/williamwen42/351/head 2025-12-04T09:23:40.0495347Z * [new branch] gh/williamwen42/351/orig -> origin/gh/williamwen42/351/orig 2025-12-04T09:23:40.0495489Z * [new branch] gh/williamwen42/352/base -> origin/gh/williamwen42/352/base 2025-12-04T09:23:40.0495638Z * [new branch] gh/williamwen42/352/head -> origin/gh/williamwen42/352/head 2025-12-04T09:23:40.0495798Z * [new branch] gh/williamwen42/352/orig -> origin/gh/williamwen42/352/orig 2025-12-04T09:23:40.0495939Z * [new branch] gh/williamwen42/353/base -> origin/gh/williamwen42/353/base 2025-12-04T09:23:40.0496084Z * [new branch] gh/williamwen42/353/head -> origin/gh/williamwen42/353/head 2025-12-04T09:23:40.0496227Z * [new branch] gh/williamwen42/353/orig -> origin/gh/williamwen42/353/orig 2025-12-04T09:23:40.0496367Z * [new branch] gh/williamwen42/354/base -> origin/gh/williamwen42/354/base 2025-12-04T09:23:40.0496510Z * [new branch] gh/williamwen42/354/head -> origin/gh/williamwen42/354/head 2025-12-04T09:23:40.0496647Z * [new branch] gh/williamwen42/354/orig -> origin/gh/williamwen42/354/orig 2025-12-04T09:23:40.0496790Z * [new branch] gh/williamwen42/355/base -> origin/gh/williamwen42/355/base 2025-12-04T09:23:40.0496938Z * [new branch] gh/williamwen42/355/head -> origin/gh/williamwen42/355/head 2025-12-04T09:23:40.0497075Z * [new branch] gh/williamwen42/355/orig -> origin/gh/williamwen42/355/orig 2025-12-04T09:23:40.0501820Z * [new branch] gh/williamwen42/356/base -> origin/gh/williamwen42/356/base 2025-12-04T09:23:40.0502208Z * [new branch] gh/williamwen42/356/head -> origin/gh/williamwen42/356/head 2025-12-04T09:23:40.0502462Z * [new branch] gh/williamwen42/356/orig -> origin/gh/williamwen42/356/orig 2025-12-04T09:23:40.0502635Z * [new branch] gh/williamwen42/357/base -> origin/gh/williamwen42/357/base 2025-12-04T09:23:40.0502903Z * [new branch] gh/williamwen42/357/head -> origin/gh/williamwen42/357/head 2025-12-04T09:23:40.0503599Z * [new branch] gh/williamwen42/357/orig -> origin/gh/williamwen42/357/orig 2025-12-04T09:23:40.0503974Z * [new branch] gh/williamwen42/358/base -> origin/gh/williamwen42/358/base 2025-12-04T09:23:40.0504301Z * [new branch] gh/williamwen42/358/head -> origin/gh/williamwen42/358/head 2025-12-04T09:23:40.0504509Z * [new branch] gh/williamwen42/358/orig -> origin/gh/williamwen42/358/orig 2025-12-04T09:23:40.0504669Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-12-04T09:23:40.0504961Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-12-04T09:23:40.0505373Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-12-04T09:23:40.0505547Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-12-04T09:23:40.0505680Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-12-04T09:23:40.0505813Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-12-04T09:23:40.0505958Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-12-04T09:23:40.0506282Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-12-04T09:23:40.0508424Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-12-04T09:23:40.0508738Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-12-04T09:23:40.0509120Z * [new branch] gh/xmfan/301/base -> origin/gh/xmfan/301/base 2025-12-04T09:23:40.0509394Z * [new branch] gh/xmfan/301/head -> origin/gh/xmfan/301/head 2025-12-04T09:23:40.0509653Z * [new branch] gh/xmfan/301/orig -> origin/gh/xmfan/301/orig 2025-12-04T09:23:40.0511963Z * [new branch] gh/xmfan/304/base -> origin/gh/xmfan/304/base 2025-12-04T09:23:40.0512129Z * [new branch] gh/xmfan/304/head -> origin/gh/xmfan/304/head 2025-12-04T09:23:40.0512288Z * [new branch] gh/xmfan/304/orig -> origin/gh/xmfan/304/orig 2025-12-04T09:23:40.0512780Z * [new branch] gh/xmfan/309/base -> origin/gh/xmfan/309/base 2025-12-04T09:23:40.0514605Z * [new branch] gh/xmfan/309/head -> origin/gh/xmfan/309/head 2025-12-04T09:23:40.0514942Z * [new branch] gh/xmfan/309/orig -> origin/gh/xmfan/309/orig 2025-12-04T09:23:40.0515125Z * [new branch] gh/xmfan/310/base -> origin/gh/xmfan/310/base 2025-12-04T09:23:40.0518344Z * [new branch] gh/xmfan/310/head -> origin/gh/xmfan/310/head 2025-12-04T09:23:40.0518519Z * [new branch] gh/xmfan/310/orig -> origin/gh/xmfan/310/orig 2025-12-04T09:23:40.0518668Z * [new branch] gh/xmfan/311/base -> origin/gh/xmfan/311/base 2025-12-04T09:23:40.0518818Z * [new branch] gh/xmfan/311/head -> origin/gh/xmfan/311/head 2025-12-04T09:23:40.0518965Z * [new branch] gh/xmfan/311/orig -> origin/gh/xmfan/311/orig 2025-12-04T09:23:40.0520813Z * [new branch] gh/xmfan/312/base -> origin/gh/xmfan/312/base 2025-12-04T09:23:40.0521226Z * [new branch] gh/xmfan/312/head -> origin/gh/xmfan/312/head 2025-12-04T09:23:40.0521841Z * [new branch] gh/xmfan/312/orig -> origin/gh/xmfan/312/orig 2025-12-04T09:23:40.0526117Z * [new branch] gh/xmfan/313/base -> origin/gh/xmfan/313/base 2025-12-04T09:23:40.0526304Z * [new branch] gh/xmfan/313/head -> origin/gh/xmfan/313/head 2025-12-04T09:23:40.0526465Z * [new branch] gh/xmfan/313/orig -> origin/gh/xmfan/313/orig 2025-12-04T09:23:40.0526641Z * [new branch] gh/xuanzhang816/27/base -> origin/gh/xuanzhang816/27/base 2025-12-04T09:23:40.0526807Z * [new branch] gh/xuanzhang816/27/head -> origin/gh/xuanzhang816/27/head 2025-12-04T09:23:40.0527152Z * [new branch] gh/xuanzhang816/27/orig -> origin/gh/xuanzhang816/27/orig 2025-12-04T09:23:40.0528797Z * [new branch] gh/xuanzhang816/32/base -> origin/gh/xuanzhang816/32/base 2025-12-04T09:23:40.0529143Z * [new branch] gh/xuanzhang816/32/head -> origin/gh/xuanzhang816/32/head 2025-12-04T09:23:40.0529637Z * [new branch] gh/xuanzhang816/32/orig -> origin/gh/xuanzhang816/32/orig 2025-12-04T09:23:40.0532402Z * [new branch] gh/xuanzhang816/33/base -> origin/gh/xuanzhang816/33/base 2025-12-04T09:23:40.0532755Z * [new branch] gh/xuanzhang816/33/head -> origin/gh/xuanzhang816/33/head 2025-12-04T09:23:40.0532955Z * [new branch] gh/xuanzhang816/33/orig -> origin/gh/xuanzhang816/33/orig 2025-12-04T09:23:40.0533355Z * [new branch] gh/xuanzhang816/34/base -> origin/gh/xuanzhang816/34/base 2025-12-04T09:23:40.0538052Z * [new branch] gh/xuanzhang816/34/head -> origin/gh/xuanzhang816/34/head 2025-12-04T09:23:40.0538583Z * [new branch] gh/xuanzhang816/34/orig -> origin/gh/xuanzhang816/34/orig 2025-12-04T09:23:40.0538838Z * [new branch] gh/xuanzhang816/35/base -> origin/gh/xuanzhang816/35/base 2025-12-04T09:23:40.0539028Z * [new branch] gh/xuanzhang816/35/head -> origin/gh/xuanzhang816/35/head 2025-12-04T09:23:40.0539254Z * [new branch] gh/xuanzhang816/35/orig -> origin/gh/xuanzhang816/35/orig 2025-12-04T09:23:40.0539922Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-12-04T09:23:40.0540119Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-12-04T09:23:40.0540268Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-12-04T09:23:40.0541758Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-12-04T09:23:40.0541945Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-12-04T09:23:40.0544413Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-12-04T09:23:40.0544770Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-12-04T09:23:40.0544998Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-12-04T09:23:40.0545205Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-12-04T09:23:40.0545750Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-12-04T09:23:40.0546900Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-12-04T09:23:40.0547411Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-12-04T09:23:40.0551761Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-12-04T09:23:40.0551960Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-12-04T09:23:40.0552107Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-12-04T09:23:40.0552254Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-12-04T09:23:40.0552392Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-12-04T09:23:40.0552531Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-12-04T09:23:40.0557204Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-12-04T09:23:40.0557381Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-12-04T09:23:40.0557527Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-12-04T09:23:40.0557665Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-12-04T09:23:40.0557821Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-12-04T09:23:40.0557972Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-12-04T09:23:40.0558548Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-12-04T09:23:40.0558750Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-12-04T09:23:40.0559894Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-12-04T09:23:40.0562102Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-12-04T09:23:40.0562331Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-12-04T09:23:40.0564159Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-12-04T09:23:40.0564493Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-12-04T09:23:40.0564967Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-12-04T09:23:40.0565129Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-12-04T09:23:40.0565348Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-12-04T09:23:40.0565564Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-12-04T09:23:40.0567404Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-12-04T09:23:40.0567740Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-12-04T09:23:40.0567912Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-12-04T09:23:40.0570372Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-12-04T09:23:40.0570692Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-12-04T09:23:40.0570893Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-12-04T09:23:40.0571273Z * [new branch] gh/yang-yu-hang/1/base -> origin/gh/yang-yu-hang/1/base 2025-12-04T09:23:40.0575205Z * [new branch] gh/yang-yu-hang/1/head -> origin/gh/yang-yu-hang/1/head 2025-12-04T09:23:40.0575551Z * [new branch] gh/yang-yu-hang/1/orig -> origin/gh/yang-yu-hang/1/orig 2025-12-04T09:23:40.0575776Z * [new branch] gh/yang-yu-hang/2/base -> origin/gh/yang-yu-hang/2/base 2025-12-04T09:23:40.0575997Z * [new branch] gh/yang-yu-hang/2/head -> origin/gh/yang-yu-hang/2/head 2025-12-04T09:23:40.0576159Z * [new branch] gh/yang-yu-hang/2/orig -> origin/gh/yang-yu-hang/2/orig 2025-12-04T09:23:40.0580723Z * [new branch] gh/yang-yu-hang/3/base -> origin/gh/yang-yu-hang/3/base 2025-12-04T09:23:40.0581065Z * [new branch] gh/yang-yu-hang/3/head -> origin/gh/yang-yu-hang/3/head 2025-12-04T09:23:40.0581269Z * [new branch] gh/yang-yu-hang/3/orig -> origin/gh/yang-yu-hang/3/orig 2025-12-04T09:23:40.0581459Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-12-04T09:23:40.0581631Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-12-04T09:23:40.0581792Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-12-04T09:23:40.0581942Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-12-04T09:23:40.0586259Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-12-04T09:23:40.0586437Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-12-04T09:23:40.0586588Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-12-04T09:23:40.0586740Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-12-04T09:23:40.0586874Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-12-04T09:23:40.0587017Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-12-04T09:23:40.0587189Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-12-04T09:23:40.0589419Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-12-04T09:23:40.0591864Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-12-04T09:23:40.0592130Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-12-04T09:23:40.0598192Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-12-04T09:23:40.0599881Z * [new branch] gh/yangw-dev/26/base -> origin/gh/yangw-dev/26/base 2025-12-04T09:23:40.0600239Z * [new branch] gh/yangw-dev/26/head -> origin/gh/yangw-dev/26/head 2025-12-04T09:23:40.0600509Z * [new branch] gh/yangw-dev/26/orig -> origin/gh/yangw-dev/26/orig 2025-12-04T09:23:40.0600664Z * [new branch] gh/yangw-dev/27/base -> origin/gh/yangw-dev/27/base 2025-12-04T09:23:40.0600865Z * [new branch] gh/yangw-dev/27/head -> origin/gh/yangw-dev/27/head 2025-12-04T09:23:40.0601006Z * [new branch] gh/yangw-dev/27/orig -> origin/gh/yangw-dev/27/orig 2025-12-04T09:23:40.0601154Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-12-04T09:23:40.0601309Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-12-04T09:23:40.0601456Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-12-04T09:23:40.0601601Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-12-04T09:23:40.0601749Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-12-04T09:23:40.0601883Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-12-04T09:23:40.0602137Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-12-04T09:23:40.0608781Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-12-04T09:23:40.0608941Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-12-04T09:23:40.0609072Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-12-04T09:23:40.0609282Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-12-04T09:23:40.0614865Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-12-04T09:23:40.0616668Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-12-04T09:23:40.0616845Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-12-04T09:23:40.0616994Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-12-04T09:23:40.0617146Z * [new branch] gh/ydwu4/312/base -> origin/gh/ydwu4/312/base 2025-12-04T09:23:40.0617309Z * [new branch] gh/ydwu4/312/head -> origin/gh/ydwu4/312/head 2025-12-04T09:23:40.0617448Z * [new branch] gh/ydwu4/312/orig -> origin/gh/ydwu4/312/orig 2025-12-04T09:23:40.0617591Z * [new branch] gh/ydwu4/322/base -> origin/gh/ydwu4/322/base 2025-12-04T09:23:40.0617736Z * [new branch] gh/ydwu4/322/head -> origin/gh/ydwu4/322/head 2025-12-04T09:23:40.0617870Z * [new branch] gh/ydwu4/322/orig -> origin/gh/ydwu4/322/orig 2025-12-04T09:23:40.0618013Z * [new branch] gh/ydwu4/327/base -> origin/gh/ydwu4/327/base 2025-12-04T09:23:40.0618158Z * [new branch] gh/ydwu4/327/head -> origin/gh/ydwu4/327/head 2025-12-04T09:23:40.0618297Z * [new branch] gh/ydwu4/327/orig -> origin/gh/ydwu4/327/orig 2025-12-04T09:23:40.0618445Z * [new branch] gh/ydwu4/328/base -> origin/gh/ydwu4/328/base 2025-12-04T09:23:40.0623488Z * [new branch] gh/ydwu4/328/head -> origin/gh/ydwu4/328/head 2025-12-04T09:23:40.0628637Z * [new branch] gh/ydwu4/328/orig -> origin/gh/ydwu4/328/orig 2025-12-04T09:23:40.0633664Z * [new branch] gh/ydwu4/329/base -> origin/gh/ydwu4/329/base 2025-12-04T09:23:40.0638747Z * [new branch] gh/ydwu4/329/head -> origin/gh/ydwu4/329/head 2025-12-04T09:23:40.0638934Z * [new branch] gh/ydwu4/329/orig -> origin/gh/ydwu4/329/orig 2025-12-04T09:23:40.0639304Z * [new branch] gh/ydwu4/330/base -> origin/gh/ydwu4/330/base 2025-12-04T09:23:40.0639461Z * [new branch] gh/ydwu4/330/head -> origin/gh/ydwu4/330/head 2025-12-04T09:23:40.0639615Z * [new branch] gh/ydwu4/330/orig -> origin/gh/ydwu4/330/orig 2025-12-04T09:23:40.0639771Z * [new branch] gh/ydwu4/331/base -> origin/gh/ydwu4/331/base 2025-12-04T09:23:40.0639976Z * [new branch] gh/ydwu4/331/head -> origin/gh/ydwu4/331/head 2025-12-04T09:23:40.0640132Z * [new branch] gh/ydwu4/331/orig -> origin/gh/ydwu4/331/orig 2025-12-04T09:23:40.0640283Z * [new branch] gh/ydwu4/332/base -> origin/gh/ydwu4/332/base 2025-12-04T09:23:40.0640613Z * [new branch] gh/ydwu4/332/head -> origin/gh/ydwu4/332/head 2025-12-04T09:23:40.0640763Z * [new branch] gh/ydwu4/332/orig -> origin/gh/ydwu4/332/orig 2025-12-04T09:23:40.0640912Z * [new branch] gh/ydwu4/333/base -> origin/gh/ydwu4/333/base 2025-12-04T09:23:40.0641059Z * [new branch] gh/ydwu4/333/head -> origin/gh/ydwu4/333/head 2025-12-04T09:23:40.0641210Z * [new branch] gh/ydwu4/333/orig -> origin/gh/ydwu4/333/orig 2025-12-04T09:23:40.0641360Z * [new branch] gh/ydwu4/334/base -> origin/gh/ydwu4/334/base 2025-12-04T09:23:40.0641511Z * [new branch] gh/ydwu4/334/head -> origin/gh/ydwu4/334/head 2025-12-04T09:23:40.0641655Z * [new branch] gh/ydwu4/334/orig -> origin/gh/ydwu4/334/orig 2025-12-04T09:23:40.0641807Z * [new branch] gh/ydwu4/335/base -> origin/gh/ydwu4/335/base 2025-12-04T09:23:40.0641961Z * [new branch] gh/ydwu4/335/head -> origin/gh/ydwu4/335/head 2025-12-04T09:23:40.0642109Z * [new branch] gh/ydwu4/335/orig -> origin/gh/ydwu4/335/orig 2025-12-04T09:23:40.0642248Z * [new branch] gh/ydwu4/337/base -> origin/gh/ydwu4/337/base 2025-12-04T09:23:40.0642399Z * [new branch] gh/ydwu4/337/head -> origin/gh/ydwu4/337/head 2025-12-04T09:23:40.0642542Z * [new branch] gh/ydwu4/337/orig -> origin/gh/ydwu4/337/orig 2025-12-04T09:23:40.0642680Z * [new branch] gh/ydwu4/339/base -> origin/gh/ydwu4/339/base 2025-12-04T09:23:40.0642834Z * [new branch] gh/ydwu4/339/head -> origin/gh/ydwu4/339/head 2025-12-04T09:23:40.0642964Z * [new branch] gh/ydwu4/339/orig -> origin/gh/ydwu4/339/orig 2025-12-04T09:23:40.0643111Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-12-04T09:23:40.0643253Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-12-04T09:23:40.0643399Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-12-04T09:23:40.0643546Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-12-04T09:23:40.0643741Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-12-04T09:23:40.0643906Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-12-04T09:23:40.0644055Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-12-04T09:23:40.0652299Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-12-04T09:23:40.0657747Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-12-04T09:23:40.0658137Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-12-04T09:23:40.0658311Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-12-04T09:23:40.0658486Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-12-04T09:23:40.0658868Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-12-04T09:23:40.0659024Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-12-04T09:23:40.0659195Z * [new branch] gh/yushangdi/1/base -> origin/gh/yushangdi/1/base 2025-12-04T09:23:40.0659423Z * [new branch] gh/yushangdi/1/head -> origin/gh/yushangdi/1/head 2025-12-04T09:23:40.0659592Z * [new branch] gh/yushangdi/10/base -> origin/gh/yushangdi/10/base 2025-12-04T09:23:40.0659770Z * [new branch] gh/yushangdi/10/head -> origin/gh/yushangdi/10/head 2025-12-04T09:23:40.0659920Z * [new branch] gh/yushangdi/10/orig -> origin/gh/yushangdi/10/orig 2025-12-04T09:23:40.0660075Z * [new branch] gh/yushangdi/11/base -> origin/gh/yushangdi/11/base 2025-12-04T09:23:40.0660226Z * [new branch] gh/yushangdi/11/head -> origin/gh/yushangdi/11/head 2025-12-04T09:23:40.0660409Z * [new branch] gh/yushangdi/11/orig -> origin/gh/yushangdi/11/orig 2025-12-04T09:23:40.0660567Z * [new branch] gh/yushangdi/2/base -> origin/gh/yushangdi/2/base 2025-12-04T09:23:40.0660718Z * [new branch] gh/yushangdi/2/head -> origin/gh/yushangdi/2/head 2025-12-04T09:23:40.0660876Z * [new branch] gh/yushangdi/7/base -> origin/gh/yushangdi/7/base 2025-12-04T09:23:40.0661026Z * [new branch] gh/yushangdi/7/head -> origin/gh/yushangdi/7/head 2025-12-04T09:23:40.0661178Z * [new branch] gh/yushangdi/7/orig -> origin/gh/yushangdi/7/orig 2025-12-04T09:23:40.0665927Z * [new branch] gh/yushangdi/8/base -> origin/gh/yushangdi/8/base 2025-12-04T09:23:40.0669012Z * [new branch] gh/yushangdi/8/head -> origin/gh/yushangdi/8/head 2025-12-04T09:23:40.0671173Z * [new branch] gh/yushangdi/8/orig -> origin/gh/yushangdi/8/orig 2025-12-04T09:23:40.0671495Z * [new branch] gh/yushangdi/9/base -> origin/gh/yushangdi/9/base 2025-12-04T09:23:40.0677660Z * [new branch] gh/yushangdi/9/head -> origin/gh/yushangdi/9/head 2025-12-04T09:23:40.0677847Z * [new branch] gh/yushangdi/9/orig -> origin/gh/yushangdi/9/orig 2025-12-04T09:23:40.0678077Z * [new branch] gh/zklaus/19/base -> origin/gh/zklaus/19/base 2025-12-04T09:23:40.0678253Z * [new branch] gh/zklaus/19/head -> origin/gh/zklaus/19/head 2025-12-04T09:23:40.0678392Z * [new branch] gh/zklaus/19/orig -> origin/gh/zklaus/19/orig 2025-12-04T09:23:40.0678537Z * [new branch] gh/zklaus/20/base -> origin/gh/zklaus/20/base 2025-12-04T09:23:40.0678673Z * [new branch] gh/zklaus/20/head -> origin/gh/zklaus/20/head 2025-12-04T09:23:40.0678818Z * [new branch] gh/zklaus/20/orig -> origin/gh/zklaus/20/orig 2025-12-04T09:23:40.0678961Z * [new branch] gh/zklaus/21/base -> origin/gh/zklaus/21/base 2025-12-04T09:23:40.0679099Z * [new branch] gh/zklaus/21/head -> origin/gh/zklaus/21/head 2025-12-04T09:23:40.0679242Z * [new branch] gh/zklaus/21/orig -> origin/gh/zklaus/21/orig 2025-12-04T09:23:40.0679378Z * [new branch] gh/zklaus/22/base -> origin/gh/zklaus/22/base 2025-12-04T09:23:40.0679511Z * [new branch] gh/zklaus/22/head -> origin/gh/zklaus/22/head 2025-12-04T09:23:40.0679655Z * [new branch] gh/zklaus/22/orig -> origin/gh/zklaus/22/orig 2025-12-04T09:23:40.0679788Z * [new branch] gh/zklaus/23/base -> origin/gh/zklaus/23/base 2025-12-04T09:23:40.0679932Z * [new branch] gh/zklaus/23/head -> origin/gh/zklaus/23/head 2025-12-04T09:23:40.0680064Z * [new branch] gh/zklaus/23/orig -> origin/gh/zklaus/23/orig 2025-12-04T09:23:40.0680652Z * [new branch] gh/zklaus/24/base -> origin/gh/zklaus/24/base 2025-12-04T09:23:40.0680807Z * [new branch] gh/zklaus/24/head -> origin/gh/zklaus/24/head 2025-12-04T09:23:40.0680941Z * [new branch] gh/zklaus/24/orig -> origin/gh/zklaus/24/orig 2025-12-04T09:23:40.0681151Z * [new branch] gh/zou3519/1197/base -> origin/gh/zou3519/1197/base 2025-12-04T09:23:40.0681299Z * [new branch] gh/zou3519/1197/head -> origin/gh/zou3519/1197/head 2025-12-04T09:23:40.0681438Z * [new branch] gh/zou3519/1197/orig -> origin/gh/zou3519/1197/orig 2025-12-04T09:23:40.0688103Z * [new branch] gh/zou3519/1199/base -> origin/gh/zou3519/1199/base 2025-12-04T09:23:40.0691219Z * [new branch] gh/zou3519/1199/head -> origin/gh/zou3519/1199/head 2025-12-04T09:23:40.0691401Z * [new branch] gh/zou3519/1199/orig -> origin/gh/zou3519/1199/orig 2025-12-04T09:23:40.0696484Z * [new branch] gh/zou3519/1200/base -> origin/gh/zou3519/1200/base 2025-12-04T09:23:40.0699032Z * [new branch] gh/zou3519/1200/head -> origin/gh/zou3519/1200/head 2025-12-04T09:23:40.0701761Z * [new branch] gh/zou3519/1200/orig -> origin/gh/zou3519/1200/orig 2025-12-04T09:23:40.0702386Z * [new branch] gh/zou3519/1201/base -> origin/gh/zou3519/1201/base 2025-12-04T09:23:40.0702569Z * [new branch] gh/zou3519/1201/head -> origin/gh/zou3519/1201/head 2025-12-04T09:23:40.0702730Z * [new branch] gh/zou3519/1201/orig -> origin/gh/zou3519/1201/orig 2025-12-04T09:23:40.0702916Z * [new branch] gh/zou3519/1202/base -> origin/gh/zou3519/1202/base 2025-12-04T09:23:40.0703089Z * [new branch] gh/zou3519/1202/head -> origin/gh/zou3519/1202/head 2025-12-04T09:23:40.0703240Z * [new branch] gh/zou3519/1202/orig -> origin/gh/zou3519/1202/orig 2025-12-04T09:23:40.0703393Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-12-04T09:23:40.0703541Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-12-04T09:23:40.0703690Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-12-04T09:23:40.0703837Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-12-04T09:23:40.0703976Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-12-04T09:23:40.0704116Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-12-04T09:23:40.0704271Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-12-04T09:23:40.0704410Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-12-04T09:23:40.0704555Z * [new branch] gh/zpcore/13/base -> origin/gh/zpcore/13/base 2025-12-04T09:23:40.0704701Z * [new branch] gh/zpcore/13/head -> origin/gh/zpcore/13/head 2025-12-04T09:23:40.0704850Z * [new branch] gh/zpcore/13/orig -> origin/gh/zpcore/13/orig 2025-12-04T09:23:40.0704999Z * [new branch] gh/zpcore/14/base -> origin/gh/zpcore/14/base 2025-12-04T09:23:40.0705133Z * [new branch] gh/zpcore/14/head -> origin/gh/zpcore/14/head 2025-12-04T09:23:40.0705274Z * [new branch] gh/zpcore/14/orig -> origin/gh/zpcore/14/orig 2025-12-04T09:23:40.0709968Z * [new branch] gh/zpcore/15/base -> origin/gh/zpcore/15/base 2025-12-04T09:23:40.0710155Z * [new branch] gh/zpcore/15/head -> origin/gh/zpcore/15/head 2025-12-04T09:23:40.0710308Z * [new branch] gh/zpcore/15/orig -> origin/gh/zpcore/15/orig 2025-12-04T09:23:40.0710621Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-12-04T09:23:40.0710766Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-12-04T09:23:40.0710923Z * [new branch] gh/zpcore/21/base -> origin/gh/zpcore/21/base 2025-12-04T09:23:40.0711118Z * [new branch] gh/zpcore/21/head -> origin/gh/zpcore/21/head 2025-12-04T09:23:40.0711264Z * [new branch] gh/zpcore/21/orig -> origin/gh/zpcore/21/orig 2025-12-04T09:23:40.0711411Z * [new branch] gh/zpcore/22/base -> origin/gh/zpcore/22/base 2025-12-04T09:23:40.0711555Z * [new branch] gh/zpcore/22/head -> origin/gh/zpcore/22/head 2025-12-04T09:23:40.0711708Z * [new branch] gh/zpcore/22/orig -> origin/gh/zpcore/22/orig 2025-12-04T09:23:40.0712424Z * [new branch] gh/zpcore/23/base -> origin/gh/zpcore/23/base 2025-12-04T09:23:40.0712624Z * [new branch] gh/zpcore/23/head -> origin/gh/zpcore/23/head 2025-12-04T09:23:40.0712778Z * [new branch] gh/zpcore/23/orig -> origin/gh/zpcore/23/orig 2025-12-04T09:23:40.0713370Z * [new branch] gh/zpcore/24/base -> origin/gh/zpcore/24/base 2025-12-04T09:23:40.0713927Z * [new branch] gh/zpcore/24/head -> origin/gh/zpcore/24/head 2025-12-04T09:23:40.0717598Z * [new branch] gh/zpcore/24/orig -> origin/gh/zpcore/24/orig 2025-12-04T09:23:40.0717794Z * [new branch] gh/zpcore/25/base -> origin/gh/zpcore/25/base 2025-12-04T09:23:40.0718494Z * [new branch] gh/zpcore/25/head -> origin/gh/zpcore/25/head 2025-12-04T09:23:40.0718649Z * [new branch] gh/zpcore/25/orig -> origin/gh/zpcore/25/orig 2025-12-04T09:23:40.0718792Z * [new branch] gh/zpcore/26/base -> origin/gh/zpcore/26/base 2025-12-04T09:23:40.0718951Z * [new branch] gh/zpcore/26/head -> origin/gh/zpcore/26/head 2025-12-04T09:23:40.0719859Z * [new branch] gh/zpcore/26/orig -> origin/gh/zpcore/26/orig 2025-12-04T09:23:40.0721079Z * [new branch] gh/zpcore/27/base -> origin/gh/zpcore/27/base 2025-12-04T09:23:40.0721260Z * [new branch] gh/zpcore/27/head -> origin/gh/zpcore/27/head 2025-12-04T09:23:40.0722821Z * [new branch] gh/zpcore/27/orig -> origin/gh/zpcore/27/orig 2025-12-04T09:23:40.0726597Z * [new branch] gh/zpcore/28/base -> origin/gh/zpcore/28/base 2025-12-04T09:23:40.0726775Z * [new branch] gh/zpcore/28/head -> origin/gh/zpcore/28/head 2025-12-04T09:23:40.0731792Z * [new branch] gh/zpcore/28/orig -> origin/gh/zpcore/28/orig 2025-12-04T09:23:40.0731983Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-12-04T09:23:40.0732151Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-12-04T09:23:40.0732286Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-12-04T09:23:40.0732418Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-12-04T09:23:40.0732569Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-12-04T09:23:40.0732704Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-12-04T09:23:40.0732848Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-12-04T09:23:40.0732979Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-12-04T09:23:40.0733108Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-12-04T09:23:40.0737949Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-12-04T09:23:40.0738305Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-12-04T09:23:40.0738454Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-12-04T09:23:40.0738597Z * [new branch] google-main -> origin/google-main 2025-12-04T09:23:40.0738819Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-12-04T09:23:40.0739418Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-12-04T09:23:40.0739678Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-12-04T09:23:40.0739902Z * [new branch] hameerabbasi/complex_tensor_subclass -> origin/hameerabbasi/complex_tensor_subclass 2025-12-04T09:23:40.0740151Z * [new branch] hameerabbasi/fix-ctensor-gradcheck-tests -> origin/hameerabbasi/fix-ctensor-gradcheck-tests 2025-12-04T09:23:40.0740362Z * [new branch] hameerabbasi/gradcheck-allclose -> origin/hameerabbasi/gradcheck-allclose 2025-12-04T09:23:40.0745729Z * [new branch] hc_baseline -> origin/hc_baseline 2025-12-04T09:23:40.0745889Z * [new branch] hhh_rand -> origin/hhh_rand 2025-12-04T09:23:40.0746024Z * [new branch] huba/f1 -> origin/huba/f1 2025-12-04T09:23:40.0746406Z * [new branch] increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test -> origin/increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test 2025-12-04T09:23:40.0746535Z * [new branch] inlining -> origin/inlining 2025-12-04T09:23:40.0753899Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-12-04T09:23:40.0759216Z * [new branch] install-torchao-0.13.0 -> origin/install-torchao-0.13.0 2025-12-04T09:23:40.0759617Z * [new branch] instrument-trunk-pull-linux-with-job-test-filters -> origin/instrument-trunk-pull-linux-with-job-test-filters 2025-12-04T09:23:40.0759794Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-12-04T09:23:40.0759984Z * [new branch] issue#58739 -> origin/issue#58739 2025-12-04T09:23:40.0760155Z * [new branch] jainapurva-patch-1 -> origin/jainapurva-patch-1 2025-12-04T09:23:40.0760297Z * [new branch] jathu/o3 -> origin/jathu/o3 2025-12-04T09:23:40.0760547Z * [new branch] jathu/sve -> origin/jathu/sve 2025-12-04T09:23:40.0760792Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-12-04T09:23:40.0761020Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-12-04T09:23:40.0761235Z * [new branch] jiannanWang/memorysnapshot_filter -> origin/jiannanWang/memorysnapshot_filter 2025-12-04T09:23:40.0761457Z * [new branch] jiannanWang/profilerstepwarning -> origin/jiannanWang/profilerstepwarning 2025-12-04T09:23:40.0761647Z * [new branch] jithunnair-amd-patch-1 -> origin/jithunnair-amd-patch-1 2025-12-04T09:23:40.0761805Z * [new branch] jithunnair-amd-patch-10 -> origin/jithunnair-amd-patch-10 2025-12-04T09:23:40.0761959Z * [new branch] jithunnair-amd-patch-2 -> origin/jithunnair-amd-patch-2 2025-12-04T09:23:40.0762117Z * [new branch] jithunnair-amd-patch-3 -> origin/jithunnair-amd-patch-3 2025-12-04T09:23:40.0762264Z * [new branch] jithunnair-amd-patch-4 -> origin/jithunnair-amd-patch-4 2025-12-04T09:23:40.0762420Z * [new branch] jithunnair-amd-patch-5 -> origin/jithunnair-amd-patch-5 2025-12-04T09:23:40.0762568Z * [new branch] jithunnair-amd-patch-6 -> origin/jithunnair-amd-patch-6 2025-12-04T09:23:40.0766673Z * [new branch] jithunnair-amd-patch-7 -> origin/jithunnair-amd-patch-7 2025-12-04T09:23:40.0767193Z * [new branch] jithunnair-amd-patch-8 -> origin/jithunnair-amd-patch-8 2025-12-04T09:23:40.0767362Z * [new branch] jithunnair-amd-patch-9 -> origin/jithunnair-amd-patch-9 2025-12-04T09:23:40.0767532Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-12-04T09:23:40.0767758Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-12-04T09:23:40.0767891Z * [new branch] kainan_test -> origin/kainan_test 2025-12-04T09:23:40.0768057Z * [new branch] larryliu0820-patch-1 -> origin/larryliu0820-patch-1 2025-12-04T09:23:40.0768254Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-12-04T09:23:40.0768459Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-12-04T09:23:40.0768610Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-12-04T09:23:40.0774130Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-12-04T09:23:40.0774483Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-12-04T09:23:40.0774668Z * [new branch] llama4-stable -> origin/llama4-stable 2025-12-04T09:23:40.0774926Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-12-04T09:23:40.0775092Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-12-04T09:23:40.0775331Z * [new branch] lucaskabela/fix_164876 -> origin/lucaskabela/fix_164876 2025-12-04T09:23:40.0776218Z * [new branch] lucaskabela/flop_counter -> origin/lucaskabela/flop_counter 2025-12-04T09:23:40.0776420Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-12-04T09:23:40.0776770Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-12-04T09:23:40.0777027Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-12-04T09:23:40.0777376Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-12-04T09:23:40.0777627Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-12-04T09:23:40.0777915Z * [new branch] lucaskabela/rnn_decomp -> origin/lucaskabela/rnn_decomp 2025-12-04T09:23:40.0784227Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-12-04T09:23:40.0784664Z * [new branch] lucaskabela/typing_ctx_manager -> origin/lucaskabela/typing_ctx_manager 2025-12-04T09:23:40.0785044Z * [new branch] lucaskabela/typing_nn_module -> origin/lucaskabela/typing_nn_module 2025-12-04T09:23:40.0785363Z * [new branch] lucaskabela/typing_user_defined -> origin/lucaskabela/typing_user_defined 2025-12-04T09:23:40.0786037Z * [new branch] lucaskabela/typing_variables -> origin/lucaskabela/typing_variables 2025-12-04T09:23:40.0786306Z * [new branch] lucaskabela/typing_variables_dicts -> origin/lucaskabela/typing_variables_dicts 2025-12-04T09:23:40.0786538Z * [new branch] lucaskabela/typing_variables_functions -> origin/lucaskabela/typing_variables_functions 2025-12-04T09:23:40.0786736Z * [new branch] lucaskabela/typing_variables_lists -> origin/lucaskabela/typing_variables_lists 2025-12-04T09:23:40.0786898Z * [new branch] lw/torch_box_by_ref -> origin/lw/torch_box_by_ref 2025-12-04T09:23:40.0787023Z * [new branch] main -> origin/main 2025-12-04T09:23:40.0787331Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2025-12-04T09:23:40.0787473Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2025-12-04T09:23:40.0787604Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2025-12-04T09:23:40.0787791Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2025-12-04T09:23:40.0792887Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2025-12-04T09:23:40.0793234Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2025-12-04T09:23:40.0793420Z * [new branch] malfet-patch-7 -> origin/malfet-patch-7 2025-12-04T09:23:40.0793564Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-12-04T09:23:40.0793830Z * [new branch] malfet/add-3.14-ci -> origin/malfet/add-3.14-ci 2025-12-04T09:23:40.0798701Z * [new branch] malfet/be-do-not-make-typos-in-build-artifacts -> origin/malfet/be-do-not-make-typos-in-build-artifacts 2025-12-04T09:23:40.0799134Z * [new branch] malfet/be-move-more-settings-to-checkout-pytorch -> origin/malfet/be-move-more-settings-to-checkout-pytorch 2025-12-04T09:23:40.0799391Z * [new branch] malfet/be-remove-misisng-neon-headers -> origin/malfet/be-remove-misisng-neon-headers 2025-12-04T09:23:40.0799589Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-12-04T09:23:40.0799804Z * [new branch] manuel/aoti_metal_shimify-thread_safe -> origin/manuel/aoti_metal_shimify-thread_safe 2025-12-04T09:23:40.0799972Z * [new branch] manuel/inductor_link_openmp -> origin/manuel/inductor_link_openmp 2025-12-04T09:23:40.0800134Z * [new branch] masnesral/metaconda -> origin/masnesral/metaconda 2025-12-04T09:23:40.0800289Z * [new branch] mem_profiler_flaky_fix -> origin/mem_profiler_flaky_fix 2025-12-04T09:23:40.0800639Z * [new branch] mem_profiler_stack_trace -> origin/mem_profiler_stack_trace 2025-12-04T09:23:40.0800802Z * [new branch] memory_profiler_stack -> origin/memory_profiler_stack 2025-12-04T09:23:40.0800963Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-12-04T09:23:40.0801117Z * [new branch] mingw_posix -> origin/mingw_posix 2025-12-04T09:23:40.0801272Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-12-04T09:23:40.0801714Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-12-04T09:23:40.0801854Z * [new branch] mlazos/acts -> origin/mlazos/acts 2025-12-04T09:23:40.0804929Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-12-04T09:23:40.0809359Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-12-04T09:23:40.0809718Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-12-04T09:23:40.0816307Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-12-04T09:23:40.0821820Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-12-04T09:23:40.0822155Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-12-04T09:23:40.0822320Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-12-04T09:23:40.0822536Z * [new branch] mlazos/bwd -> origin/mlazos/bwd 2025-12-04T09:23:40.0822710Z * [new branch] mlazos/combo-test -> origin/mlazos/combo-test 2025-12-04T09:23:40.0822880Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-12-04T09:23:40.0823137Z * [new branch] mlazos/cuda-cmd-log -> origin/mlazos/cuda-cmd-log 2025-12-04T09:23:40.0823600Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-12-04T09:23:40.0823835Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-12-04T09:23:40.0823994Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-12-04T09:23:40.0824211Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-12-04T09:23:40.0824376Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-12-04T09:23:40.0824521Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-12-04T09:23:40.0824666Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-12-04T09:23:40.0824807Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-12-04T09:23:40.0824962Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-12-04T09:23:40.0825102Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-12-04T09:23:40.0825252Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-12-04T09:23:40.0825382Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-12-04T09:23:40.0825559Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-12-04T09:23:40.0825707Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-12-04T09:23:40.0825841Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-12-04T09:23:40.0825986Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-12-04T09:23:40.0826145Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-12-04T09:23:40.0826293Z * [new branch] mlazos/fp8-fixes -> origin/mlazos/fp8-fixes 2025-12-04T09:23:40.0826432Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-12-04T09:23:40.0826569Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-12-04T09:23:40.0826717Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-12-04T09:23:40.0826873Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-12-04T09:23:40.0826997Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-12-04T09:23:40.0827277Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-12-04T09:23:40.0827423Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-12-04T09:23:40.0827653Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-12-04T09:23:40.0827804Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-12-04T09:23:40.0830019Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-12-04T09:23:40.0830359Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-12-04T09:23:40.0830509Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-12-04T09:23:40.0830816Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-12-04T09:23:40.0832417Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-12-04T09:23:40.0832576Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-12-04T09:23:40.0832984Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-12-04T09:23:40.0836175Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-12-04T09:23:40.0836346Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-12-04T09:23:40.0836689Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-12-04T09:23:40.0836815Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-12-04T09:23:40.0836946Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-12-04T09:23:40.0837446Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-12-04T09:23:40.0838980Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-12-04T09:23:40.0839470Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-12-04T09:23:40.0839654Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-12-04T09:23:40.0840053Z * [new branch] mlazos/inductor-streams -> origin/mlazos/inductor-streams 2025-12-04T09:23:40.0841662Z * [new branch] mlazos/main -> origin/mlazos/main 2025-12-04T09:23:40.0841976Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-12-04T09:23:40.0842265Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-12-04T09:23:40.0845805Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-12-04T09:23:40.0846043Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-12-04T09:23:40.0846203Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-12-04T09:23:40.0846347Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-12-04T09:23:40.0846521Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-12-04T09:23:40.0847324Z * [new branch] mlazos/overguarding -> origin/mlazos/overguarding 2025-12-04T09:23:40.0853285Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-12-04T09:23:40.0853659Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-12-04T09:23:40.0860542Z * [new branch] mlazos/resnet-fix -> origin/mlazos/resnet-fix 2025-12-04T09:23:40.0860732Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-12-04T09:23:40.0860908Z * [new branch] mlazos/rm-code -> origin/mlazos/rm-code 2025-12-04T09:23:40.0861045Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-12-04T09:23:40.0861178Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-12-04T09:23:40.0861375Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-12-04T09:23:40.0862357Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-12-04T09:23:40.0862510Z * [new branch] mlazos/stests -> origin/mlazos/stests 2025-12-04T09:23:40.0862671Z * [new branch] mlazos/stream-ops -> origin/mlazos/stream-ops 2025-12-04T09:23:40.0862825Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-12-04T09:23:40.0862997Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-12-04T09:23:40.0863130Z * [new branch] mlazos/test -> origin/mlazos/test 2025-12-04T09:23:40.0869443Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-12-04T09:23:40.0869649Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-12-04T09:23:40.0869817Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-12-04T09:23:40.0869970Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-12-04T09:23:40.0870124Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-12-04T09:23:40.0870497Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-12-04T09:23:40.0870648Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-12-04T09:23:40.0870851Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-12-04T09:23:40.0871093Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-12-04T09:23:40.0871251Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-12-04T09:23:40.0871416Z * [new branch] mlazos/user-stream-base -> origin/mlazos/user-stream-base 2025-12-04T09:23:40.0871566Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-12-04T09:23:40.0877592Z * [new branch] mlazos/user-streams-backup -> origin/mlazos/user-streams-backup 2025-12-04T09:23:40.0877818Z * [new branch] mlazos/user-streams-backup2 -> origin/mlazos/user-streams-backup2 2025-12-04T09:23:40.0877988Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-12-04T09:23:40.0878144Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-12-04T09:23:40.0878298Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-12-04T09:23:40.0878461Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-12-04T09:23:40.0878598Z * [new branch] module-shim -> origin/module-shim 2025-12-04T09:23:40.0878725Z * [new branch] move_config -> origin/move_config 2025-12-04T09:23:40.0878875Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-12-04T09:23:40.0879192Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-12-04T09:23:40.0880833Z * [new branch] mwizak/fix-triton-block-shape -> origin/mwizak/fix-triton-block-shape 2025-12-04T09:23:40.0881049Z * [new branch] my_varlen_backup -> origin/my_varlen_backup 2025-12-04T09:23:40.0886919Z * [new branch] nativert_num_outputs -> origin/nativert_num_outputs 2025-12-04T09:23:40.0891742Z * [new branch] new-codegen -> origin/new-codegen 2025-12-04T09:23:40.0896073Z * [new branch] newtest-base -> origin/newtest-base 2025-12-04T09:23:40.0896431Z * [new branch] ngimel/addmm_dtype -> origin/ngimel/addmm_dtype 2025-12-04T09:23:40.0896623Z * [new branch] ngimel/div_inv -> origin/ngimel/div_inv 2025-12-04T09:23:40.0896788Z * [new branch] ngimel/error_index_list -> origin/ngimel/error_index_list 2025-12-04T09:23:40.0896926Z * [new branch] ngimel/gather_grid -> origin/ngimel/gather_grid 2025-12-04T09:23:40.0897111Z * [new branch] ngimel/gather_grid_release -> origin/ngimel/gather_grid_release 2025-12-04T09:23:40.0897253Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-12-04T09:23:40.0897405Z * [new branch] ngimel/hostalloc -> origin/ngimel/hostalloc 2025-12-04T09:23:40.0897554Z * [new branch] ngimel/storage_id -> origin/ngimel/storage_id 2025-12-04T09:23:40.0897683Z * [new branch] nightly -> origin/nightly 2025-12-04T09:23:40.0897899Z * [new branch] nikitaved/addmm_1_rowcol_lt_path_check -> origin/nikitaved/addmm_1_rowcol_lt_path_check 2025-12-04T09:23:40.0898134Z * [new branch] nikitaved/addmm_epilogue_fusions_2d_bias -> origin/nikitaved/addmm_epilogue_fusions_2d_bias 2025-12-04T09:23:40.0898370Z * [new branch] nikitaved/addmm_epilogue_fusions_inductor -> origin/nikitaved/addmm_epilogue_fusions_inductor 2025-12-04T09:23:40.0898604Z * [new branch] nikitaved/addmm_epilogue_fusions_scratch -> origin/nikitaved/addmm_epilogue_fusions_scratch 2025-12-04T09:23:40.0898992Z * [new branch] nikitaved/grad_addmm_epilogue_fusions -> origin/nikitaved/grad_addmm_epilogue_fusions 2025-12-04T09:23:40.0899217Z * [new branch] nikitaved/simpler_can_use_32bit_index -> origin/nikitaved/simpler_can_use_32bit_index 2025-12-04T09:23:40.0899447Z * [new branch] nikitaved/test -> origin/nikitaved/test 2025-12-04T09:23:40.0899696Z * [new branch] nmacchioni-perf-test-async-autotune -> origin/nmacchioni-perf-test-async-autotune 2025-12-04T09:23:40.0899856Z * [new branch] no_distributed_log_spew -> origin/no_distributed_log_spew 2025-12-04T09:23:40.0899986Z * [new branch] nofun-hack -> origin/nofun-hack 2025-12-04T09:23:40.0900107Z * [new branch] norm_bench -> origin/norm_bench 2025-12-04T09:23:40.0900263Z * [new branch] nullplay/fuse_matmul -> origin/nullplay/fuse_matmul 2025-12-04T09:23:40.0900462Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-12-04T09:23:40.0900737Z * [new branch] optimizer_test -> origin/optimizer_test 2025-12-04T09:23:40.0902462Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-12-04T09:23:40.0902641Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-12-04T09:23:40.0905206Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-12-04T09:23:40.0905369Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-12-04T09:23:40.0905504Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-12-04T09:23:40.0905992Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-12-04T09:23:40.0907625Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-12-04T09:23:40.0913300Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-12-04T09:23:40.0915979Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-12-04T09:23:40.0916276Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-12-04T09:23:40.0916459Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-12-04T09:23:40.0916596Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-12-04T09:23:40.0916804Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-12-04T09:23:40.0916954Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-12-04T09:23:40.0917114Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-12-04T09:23:40.0917260Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-12-04T09:23:40.0917483Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-12-04T09:23:40.0918147Z * [new branch] orig/release/2.9 -> origin/orig/release/2.9 2025-12-04T09:23:40.0918366Z * [new branch] origin/gh/fxdawnn/1/base -> origin/origin/gh/fxdawnn/1/base 2025-12-04T09:23:40.0918640Z * [new branch] origin/gh/fxdawnn/1/orig -> origin/origin/gh/fxdawnn/1/orig 2025-12-04T09:23:40.0918812Z * [new branch] origin/gh/zpcore/14/orig -> origin/origin/gh/zpcore/14/orig 2025-12-04T09:23:40.0919234Z * [new branch] oulgen-patch-1 -> origin/oulgen-patch-1 2025-12-04T09:23:40.0920462Z * [new branch] oulgen-patch-2 -> origin/oulgen-patch-2 2025-12-04T09:23:40.0921044Z * [new branch] oulgen-patch-3 -> origin/oulgen-patch-3 2025-12-04T09:23:40.0928155Z * [new branch] oulgen-patch-4 -> origin/oulgen-patch-4 2025-12-04T09:23:40.0928507Z * [new branch] padded-tensor -> origin/padded-tensor 2025-12-04T09:23:40.0928645Z * [new branch] pca2 -> origin/pca2 2025-12-04T09:23:40.0928800Z * [new branch] per_channel_backup -> origin/per_channel_backup 2025-12-04T09:23:40.0928983Z * [new branch] perf_ops -> origin/perf_ops 2025-12-04T09:23:40.0929121Z * [new branch] perf_ops_2_9 -> origin/perf_ops_2_9 2025-12-04T09:23:40.0933572Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-12-04T09:23:40.0933966Z * [new branch] pianpwk/__draft_debug_mode -> origin/pianpwk/__draft_debug_mode 2025-12-04T09:23:40.0934187Z * [new branch] pianpwk/_debug_mode_for_triton_draft -> origin/pianpwk/_debug_mode_for_triton_draft 2025-12-04T09:23:40.0934386Z * [new branch] pianpwk/_debug_nn_module_compile -> origin/pianpwk/_debug_nn_module_compile 2025-12-04T09:23:40.0934579Z * [new branch] pianpwk/_draft_triton_11_3 -> origin/pianpwk/_draft_triton_11_3 2025-12-04T09:23:40.0934755Z * [new branch] pianpwk/_manual_bucket_draft -> origin/pianpwk/_manual_bucket_draft 2025-12-04T09:23:40.0935000Z * [new branch] pianpwk/_profile_w_dispatch_keys -> origin/pianpwk/_profile_w_dispatch_keys 2025-12-04T09:23:40.0935191Z * [new branch] pianpwk/_super_draft_debug_mode -> origin/pianpwk/_super_draft_debug_mode 2025-12-04T09:23:40.0935411Z * [new branch] pianpwk/_unbacked_local_shard_size -> origin/pianpwk/_unbacked_local_shard_size 2025-12-04T09:23:40.0935569Z * [new branch] pianpwk/anomaly_tb -> origin/pianpwk/anomaly_tb 2025-12-04T09:23:40.0935733Z * [new branch] pianpwk/auto_fx_annotate -> origin/pianpwk/auto_fx_annotate 2025-12-04T09:23:40.0935952Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-12-04T09:23:40.0936120Z * [new branch] pianpwk/bert_dynamic_perf -> origin/pianpwk/bert_dynamic_perf 2025-12-04T09:23:40.0939440Z * [new branch] pianpwk/debug_fwd_stack_traces -> origin/pianpwk/debug_fwd_stack_traces 2025-12-04T09:23:40.0939706Z * [new branch] pianpwk/debug_hash_tensor -> origin/pianpwk/debug_hash_tensor 2025-12-04T09:23:40.0939891Z * [new branch] pianpwk/debug_mode_annotate -> origin/pianpwk/debug_mode_annotate 2025-12-04T09:23:40.0940182Z * [new branch] pianpwk/debug_mode_defaults -> origin/pianpwk/debug_mode_defaults 2025-12-04T09:23:40.0940365Z * [new branch] pianpwk/debug_mode_hacks -> origin/pianpwk/debug_mode_hacks 2025-12-04T09:23:40.0940688Z * [new branch] pianpwk/debug_mode_opcall_refactor -> origin/pianpwk/debug_mode_opcall_refactor 2025-12-04T09:23:40.0946685Z * [new branch] pianpwk/debug_mode_show_ids -> origin/pianpwk/debug_mode_show_ids 2025-12-04T09:23:40.0951718Z * [new branch] pianpwk/debug_mode_triton -> origin/pianpwk/debug_mode_triton 2025-12-04T09:23:40.0957110Z * [new branch] pianpwk/debug_show_stack_trace -> origin/pianpwk/debug_show_stack_trace 2025-12-04T09:23:40.0959163Z * [new branch] pianpwk/debug_wait_on_collective -> origin/pianpwk/debug_wait_on_collective 2025-12-04T09:23:40.0959420Z * [new branch] pianpwk/debugmode_compile_tf -> origin/pianpwk/debugmode_compile_tf 2025-12-04T09:23:40.0959661Z * [new branch] pianpwk/dispatch_key_debugging_for_debug -> origin/pianpwk/dispatch_key_debugging_for_debug 2025-12-04T09:23:40.0959873Z * [new branch] pianpwk/draft_debug_mode_tfcompile -> origin/pianpwk/draft_debug_mode_tfcompile 2025-12-04T09:23:40.0960077Z * [new branch] pianpwk/draft_multikernel_nn -> origin/pianpwk/draft_multikernel_nn 2025-12-04T09:23:40.0960290Z * [new branch] pianpwk/draft_multikernel_status_10_5 -> origin/pianpwk/draft_multikernel_status_10_5 2025-12-04T09:23:40.0960841Z * [new branch] pianpwk/dtensor_custom_chunk -> origin/pianpwk/dtensor_custom_chunk 2025-12-04T09:23:40.0961032Z * [new branch] pianpwk/dtensor_unbacked_keypath -> origin/pianpwk/dtensor_unbacked_keypath 2025-12-04T09:23:40.0961260Z * [new branch] pianpwk/event_list_tree -> origin/pianpwk/event_list_tree 2025-12-04T09:23:40.0961425Z * [new branch] pianpwk/false_numel_refs -> origin/pianpwk/false_numel_refs 2025-12-04T09:23:40.0961589Z * [new branch] pianpwk/maybe_guard_rel -> origin/pianpwk/maybe_guard_rel 2025-12-04T09:23:40.0961778Z * [new branch] pianpwk/multikernel_hints_draft -> origin/pianpwk/multikernel_hints_draft 2025-12-04T09:23:40.0961971Z * [new branch] pianpwk/no_size_oblivious_slice_scat -> origin/pianpwk/no_size_oblivious_slice_scat 2025-12-04T09:23:40.0962192Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-12-04T09:23:40.0962347Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-12-04T09:23:40.0962532Z * [new branch] pianpwk/skip_python_keys_alternate -> origin/pianpwk/skip_python_keys_alternate 2025-12-04T09:23:40.0962721Z * [new branch] pianpwk/skip_python_keys_in_guards -> origin/pianpwk/skip_python_keys_in_guards 2025-12-04T09:23:40.0962875Z * [new branch] pianpwk/sym_tokens_draft -> origin/pianpwk/sym_tokens_draft 2025-12-04T09:23:40.0963039Z * [new branch] pianpwk/symint_one_hot -> origin/pianpwk/symint_one_hot 2025-12-04T09:23:40.0963242Z * [new branch] pianpwk/test_pointwise_guard_or_false -> origin/pianpwk/test_pointwise_guard_or_false 2025-12-04T09:23:40.0963423Z * [new branch] pianpwk/totally_draft_sym_wrap -> origin/pianpwk/totally_draft_sym_wrap 2025-12-04T09:23:40.0963582Z * [new branch] pianpwk/try_dumb_stuff -> origin/pianpwk/try_dumb_stuff 2025-12-04T09:23:40.0963734Z * [new branch] pianpwk/try_dumb_stuff_2 -> origin/pianpwk/try_dumb_stuff_2 2025-12-04T09:23:40.0963913Z * [new branch] pianpwk/unbacked_dtensor_mm -> origin/pianpwk/unbacked_dtensor_mm 2025-12-04T09:23:40.0964094Z * [new branch] pianpwk/unbacked_tracing_12_2 -> origin/pianpwk/unbacked_tracing_12_2 2025-12-04T09:23:40.0964247Z * [new branch] pianpwk/user_symints -> origin/pianpwk/user_symints 2025-12-04T09:23:40.0964405Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-12-04T09:23:40.0966058Z * [new branch] piz/fix_partial_backward_1112 -> origin/piz/fix_partial_backward_1112 2025-12-04T09:23:40.0966619Z * [new branch] piz/prop_cache_clean -> origin/piz/prop_cache_clean 2025-12-04T09:23:40.0966804Z * [new branch] pool-separate -> origin/pool-separate 2025-12-04T09:23:40.0966969Z * [new branch] pr-156087 -> origin/pr-156087 2025-12-04T09:23:40.0973385Z * [new branch] pr/131860 -> origin/pr/131860 2025-12-04T09:23:40.0978472Z * [new branch] predispatch_to -> origin/predispatch_to 2025-12-04T09:23:40.0982835Z * [new branch] protect-c17 -> origin/protect-c17 2025-12-04T09:23:40.0987607Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-12-04T09:23:40.0992740Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-12-04T09:23:40.0997067Z * [new branch] q1l1/fix_device_moved_constant_type_unknown -> origin/q1l1/fix_device_moved_constant_type_unknown 2025-12-04T09:23:40.0997363Z * [new branch] q1l1/fix_wrong_default_type_for_kernel_call_args -> origin/q1l1/fix_wrong_default_type_for_kernel_call_args 2025-12-04T09:23:40.0997718Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-12-04T09:23:40.0997876Z * [new branch] quote-pytest_cache -> origin/quote-pytest_cache 2025-12-04T09:23:40.0998090Z * [new branch] reland-accgrad-stream-warn -> origin/reland-accgrad-stream-warn 2025-12-04T09:23:40.0998276Z * [new branch] release/1.10 -> origin/release/1.10 2025-12-04T09:23:40.0998406Z * [new branch] release/1.11 -> origin/release/1.11 2025-12-04T09:23:40.0998535Z * [new branch] release/1.12 -> origin/release/1.12 2025-12-04T09:23:40.0998655Z * [new branch] release/1.13 -> origin/release/1.13 2025-12-04T09:23:40.0998789Z * [new branch] release/1.4 -> origin/release/1.4 2025-12-04T09:23:40.0998917Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-12-04T09:23:40.0999045Z * [new branch] release/1.5 -> origin/release/1.5 2025-12-04T09:23:40.0999173Z * [new branch] release/1.6 -> origin/release/1.6 2025-12-04T09:23:40.0999295Z * [new branch] release/1.7 -> origin/release/1.7 2025-12-04T09:23:40.0999428Z * [new branch] release/1.8 -> origin/release/1.8 2025-12-04T09:23:40.0999556Z * [new branch] release/1.9 -> origin/release/1.9 2025-12-04T09:23:40.0999681Z * [new branch] release/2.0 -> origin/release/2.0 2025-12-04T09:23:40.0999811Z * [new branch] release/2.1 -> origin/release/2.1 2025-12-04T09:23:40.0999933Z * [new branch] release/2.2 -> origin/release/2.2 2025-12-04T09:23:40.1000057Z * [new branch] release/2.3 -> origin/release/2.3 2025-12-04T09:23:40.1000188Z * [new branch] release/2.4 -> origin/release/2.4 2025-12-04T09:23:40.1000522Z * [new branch] release/2.5 -> origin/release/2.5 2025-12-04T09:23:40.1000667Z * [new branch] release/2.6 -> origin/release/2.6 2025-12-04T09:23:40.1000789Z * [new branch] release/2.7 -> origin/release/2.7 2025-12-04T09:23:40.1000918Z * [new branch] release/2.8 -> origin/release/2.8 2025-12-04T09:23:40.1001047Z * [new branch] release/2.9 -> origin/release/2.9 2025-12-04T09:23:40.1001179Z * [new branch] release_notes -> origin/release_notes 2025-12-04T09:23:40.1001350Z * [new branch] remove_pyinterpreter -> origin/remove_pyinterpreter 2025-12-04T09:23:40.1001592Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-12-04T09:23:40.1001831Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-12-04T09:23:40.1002058Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-12-04T09:23:40.1002276Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-12-04T09:23:40.1002536Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-12-04T09:23:40.1002743Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-12-04T09:23:40.1002936Z * [new branch] revert-152361-gh/fadara01/1/head -> origin/revert-152361-gh/fadara01/1/head 2025-12-04T09:23:40.1003125Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-12-04T09:23:40.1003445Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-12-04T09:23:40.1003672Z * [new branch] revert-hoo-invoke-subgraph -> origin/revert-hoo-invoke-subgraph 2025-12-04T09:23:40.1003851Z * [new branch] revert_always_build_distributed -> origin/revert_always_build_distributed 2025-12-04T09:23:40.1003980Z * [new branch] rms_norm_patch -> origin/rms_norm_patch 2025-12-04T09:23:40.1008419Z * [new branch] ruisi/fix_all_to_all_estimation -> origin/ruisi/fix_all_to_all_estimation 2025-12-04T09:23:40.1008646Z * [new branch] ruisi/fix_comm_estimation -> origin/ruisi/fix_comm_estimation 2025-12-04T09:23:40.1010033Z * [new branch] ruisi/fix_dynamic_shape_estimation -> origin/ruisi/fix_dynamic_shape_estimation 2025-12-04T09:23:40.1010248Z * [new branch] ruisi/fix_llama3_autobucketing -> origin/ruisi/fix_llama3_autobucketing 2025-12-04T09:23:40.1010451Z * [new branch] ruisi/fix_manual_bucketing_ep_pass -> origin/ruisi/fix_manual_bucketing_ep_pass 2025-12-04T09:23:40.1010637Z * [new branch] ruisi/manual_bucket_pass -> origin/ruisi/manual_bucket_pass 2025-12-04T09:23:40.1010918Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-12-04T09:23:40.1011092Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-12-04T09:23:40.1011256Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-12-04T09:23:40.1011382Z * [new branch] rzou/njt -> origin/rzou/njt 2025-12-04T09:23:40.1011514Z * [new branch] rzou/pca -> origin/rzou/pca 2025-12-04T09:23:40.1011648Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-12-04T09:23:40.1015553Z * [new branch] samplevllm -> origin/samplevllm 2025-12-04T09:23:40.1015895Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-12-04T09:23:40.1016081Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-12-04T09:23:40.1016300Z * [new branch] sapling-pr-archive-tushar00jain -> origin/sapling-pr-archive-tushar00jain 2025-12-04T09:23:40.1016425Z * [new branch] save -> origin/save 2025-12-04T09:23:40.1016553Z * [new branch] scaled_mm -> origin/scaled_mm 2025-12-04T09:23:40.1021925Z * [new branch] scan_attempt -> origin/scan_attempt 2025-12-04T09:23:40.1022056Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-12-04T09:23:40.1022261Z * [new branch] sekyondaMeta-dynamoconfig-fix -> origin/sekyondaMeta-dynamoconfig-fix 2025-12-04T09:23:40.1022416Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-12-04T09:23:40.1022583Z * [new branch] shoumikhin-patch-1 -> origin/shoumikhin-patch-1 2025-12-04T09:23:40.1027388Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-12-04T09:23:40.1027879Z * [new branch] some_rocm_inductor_skips -> origin/some_rocm_inductor_skips 2025-12-04T09:23:40.1028170Z * [new branch] soulitzer/stash-tls-ac -> origin/soulitzer/stash-tls-ac 2025-12-04T09:23:40.1028366Z * [new branch] sparse-mm-bf16-support -> origin/sparse-mm-bf16-support 2025-12-04T09:23:40.1028535Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-12-04T09:23:40.1028665Z * [new branch] suo -> origin/suo 2025-12-04T09:23:40.1028801Z * [new branch] sve-poc -> origin/sve-poc 2025-12-04T09:23:40.1029359Z * [new branch] switch-bn -> origin/switch-bn 2025-12-04T09:23:40.1029732Z * [new branch] sy_annotation_in_autograd_hop -> origin/sy_annotation_in_autograd_hop 2025-12-04T09:23:40.1029903Z * [new branch] sy_aot_eager_record -> origin/sy_aot_eager_record 2025-12-04T09:23:40.1030049Z * [new branch] sy_custom_bucketing -> origin/sy_custom_bucketing 2025-12-04T09:23:40.1030287Z * [new branch] sy_debug_mode_test -> origin/sy_debug_mode_test 2025-12-04T09:23:40.1030437Z * [new branch] sy_deserialize -> origin/sy_deserialize 2025-12-04T09:23:40.1030573Z * [new branch] sy_dump_gm_code -> origin/sy_dump_gm_code 2025-12-04T09:23:40.1033079Z * [new branch] sy_exp -> origin/sy_exp 2025-12-04T09:23:40.1033430Z * [new branch] sy_export_annotation -> origin/sy_export_annotation 2025-12-04T09:23:40.1033588Z * [new branch] sy_invoke_subgraph -> origin/sy_invoke_subgraph 2025-12-04T09:23:40.1033751Z * [new branch] sy_kernel_bw_name -> origin/sy_kernel_bw_name 2025-12-04T09:23:40.1033882Z * [new branch] sy_multi_arch -> origin/sy_multi_arch 2025-12-04T09:23:40.1036709Z * [new branch] sy_nn_module_stack -> origin/sy_nn_module_stack 2025-12-04T09:23:40.1036995Z * [new branch] sy_original_dtensor -> origin/sy_original_dtensor 2025-12-04T09:23:40.1037144Z * [new branch] sy_profiler_cia -> origin/sy_profiler_cia 2025-12-04T09:23:40.1037288Z * [new branch] symm_mem_sync -> origin/symm_mem_sync 2025-12-04T09:23:40.1037466Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-12-04T09:23:40.1037657Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-12-04T09:23:40.1037849Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-12-04T09:23:40.1038671Z * [new branch] test-old -> origin/test-old 2025-12-04T09:23:40.1039855Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-12-04T09:23:40.1040680Z * [new branch] tianren/customOp_autotune_fix -> origin/tianren/customOp_autotune_fix 2025-12-04T09:23:40.1044629Z * [new branch] tianren/customOp_enable_max_autotune -> origin/tianren/customOp_enable_max_autotune 2025-12-04T09:23:40.1044923Z * [new branch] tianren/customOp_fusion -> origin/tianren/customOp_fusion 2025-12-04T09:23:40.1045161Z * [new branch] tianren/customop_collectiveop_benchmark -> origin/tianren/customop_collectiveop_benchmark 2025-12-04T09:23:40.1045419Z * [new branch] tianren/customop_collectiveop_benchmark_fix -> origin/tianren/customop_collectiveop_benchmark_fix 2025-12-04T09:23:40.1045607Z * [new branch] tianren/customop_dynamic_config -> origin/tianren/customop_dynamic_config 2025-12-04T09:23:40.1050704Z * [new branch] tianren/dynamic_range_input -> origin/tianren/dynamic_range_input 2025-12-04T09:23:40.1050945Z * [new branch] tianren/dynamic_range_input_fix -> origin/tianren/dynamic_range_input_fix 2025-12-04T09:23:40.1051181Z * [new branch] tianren/dynamic_range_input_merge -> origin/tianren/dynamic_range_input_merge 2025-12-04T09:23:40.1051380Z * [new branch] tianren/flex_paged_attn_fix_temp -> origin/tianren/flex_paged_attn_fix_temp 2025-12-04T09:23:40.1051543Z * [new branch] tianren/fx_codegen_dump -> origin/tianren/fx_codegen_dump 2025-12-04T09:23:40.1051723Z * [new branch] tianren/symmetric_memory -> origin/tianren/symmetric_memory 2025-12-04T09:23:40.1056238Z * [new branch] tianren/test -> origin/tianren/test 2025-12-04T09:23:40.1056442Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-12-04T09:23:40.1056846Z * [new branch] tmp -> origin/tmp 2025-12-04T09:23:40.1056990Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-12-04T09:23:40.1057161Z * [new branch] torchtitan_integration -> origin/torchtitan_integration 2025-12-04T09:23:40.1057405Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-12-04T09:23:40.1057580Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-12-04T09:23:40.1057726Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-12-04T09:23:40.1057856Z * [new branch] triton_kernel -> origin/triton_kernel 2025-12-04T09:23:40.1057988Z * [new branch] tt_pkg_1908 -> origin/tt_pkg_1908 2025-12-04T09:23:40.1058118Z * [new branch] type_dec -> origin/type_dec 2025-12-04T09:23:40.1058304Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-12-04T09:23:40.1063094Z * [new branch] update-audio-commit-hash/17630256502-1803-1 -> origin/update-audio-commit-hash/17630256502-1803-1 2025-12-04T09:23:40.1063387Z * [new branch] update-audio-commit-hash/19087141161-1916-1 -> origin/update-audio-commit-hash/19087141161-1916-1 2025-12-04T09:23:40.1063648Z * [new branch] update-audio-commit-hash/19250643381-1929-1 -> origin/update-audio-commit-hash/19250643381-1929-1 2025-12-04T09:23:40.1063899Z * [new branch] update-audio-commit-hash/19397724337-1935-1 -> origin/update-audio-commit-hash/19397724337-1935-1 2025-12-04T09:23:40.1064678Z * [new branch] update-audio-commit-hash/19555670148-1941-1 -> origin/update-audio-commit-hash/19555670148-1941-1 2025-12-04T09:23:40.1064915Z * [new branch] update-audio-commit-hash/19750627930-1946-1 -> origin/update-audio-commit-hash/19750627930-1946-1 2025-12-04T09:23:40.1065172Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-12-04T09:23:40.1065422Z * [new branch] update-vision-commit-hash/19087141161-1916-1 -> origin/update-vision-commit-hash/19087141161-1916-1 2025-12-04T09:23:40.1065671Z * [new branch] update-vision-commit-hash/19184897099-1925-1 -> origin/update-vision-commit-hash/19184897099-1925-1 2025-12-04T09:23:40.1069992Z * [new branch] update-vision-commit-hash/19250643381-1929-1 -> origin/update-vision-commit-hash/19250643381-1929-1 2025-12-04T09:23:40.1070439Z * [new branch] update-vision-commit-hash/19381328640-1934-1 -> origin/update-vision-commit-hash/19381328640-1934-1 2025-12-04T09:23:40.1070835Z * [new branch] update-vision-commit-hash/19485237164-1938-1 -> origin/update-vision-commit-hash/19485237164-1938-1 2025-12-04T09:23:40.1071306Z * [new branch] update-vllm-commit-hash/18451675449-1879-1 -> origin/update-vllm-commit-hash/18451675449-1879-1 2025-12-04T09:23:40.1071675Z * [new branch] update-vllm-dockerfile -> origin/update-vllm-dockerfile 2025-12-04T09:23:40.1071906Z * [new branch] update-xla-commit-hash/19224287370-211-1 -> origin/update-xla-commit-hash/19224287370-211-1 2025-12-04T09:23:40.1072146Z * [new branch] update-xla-commit-hash/19422028566-212-1 -> origin/update-xla-commit-hash/19422028566-212-1 2025-12-04T09:23:40.1072368Z * [new branch] update-xla-commit-hash/19626841311-213-1 -> origin/update-xla-commit-hash/19626841311-213-1 2025-12-04T09:23:40.1072601Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-12-04T09:23:40.1072770Z * [new branch] update_operator_readme -> origin/update_operator_readme 2025-12-04T09:23:40.1077449Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-12-04T09:23:40.1077664Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-12-04T09:23:40.1077842Z * [new branch] update_slow_tests_1762155677 -> origin/update_slow_tests_1762155677 2025-12-04T09:23:40.1078184Z * [new branch] update_slow_tests_1763365283 -> origin/update_slow_tests_1763365283 2025-12-04T09:23:40.1078415Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-12-04T09:23:40.1078578Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-12-04T09:23:40.1078760Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-12-04T09:23:40.1078963Z * [new branch] upload-tests-for-autorevert -> origin/upload-tests-for-autorevert 2025-12-04T09:23:40.1079101Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-12-04T09:23:40.1079262Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-12-04T09:23:40.1080879Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-12-04T09:23:40.1086972Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-12-04T09:23:40.1087148Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-12-04T09:23:40.1087271Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-12-04T09:23:40.1087417Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-12-04T09:23:40.1087545Z * [new branch] validate_fn -> origin/validate_fn 2025-12-04T09:23:40.1087680Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-12-04T09:23:40.1091376Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-12-04T09:23:40.1091752Z * [new branch] varlen-api -> origin/varlen-api 2025-12-04T09:23:40.1092040Z * [new branch] varlen-api-backup -> origin/varlen-api-backup 2025-12-04T09:23:40.1092338Z * [new branch] varlen_batch_invariance -> origin/varlen_batch_invariance 2025-12-04T09:23:40.1092574Z * [new branch] viable/strict -> origin/viable/strict 2025-12-04T09:23:40.1093337Z * [new branch] vishal9-team/dtensor_parallelism_toy -> origin/vishal9-team/dtensor_parallelism_toy 2025-12-04T09:23:40.1093546Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-12-04T09:23:40.1093673Z * [new branch] vllmpin -> origin/vllmpin 2025-12-04T09:23:40.1093883Z * [new branch] vscode-recommend-pyrefly -> origin/vscode-recommend-pyrefly 2025-12-04T09:23:40.1094031Z * [new branch] wdvr-patch-1 -> origin/wdvr-patch-1 2025-12-04T09:23:40.1094186Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-12-04T09:23:40.1099282Z * [new branch] whc/pei -> origin/whc/pei 2025-12-04T09:23:40.1099622Z * [new branch] whc/pp_fix -> origin/whc/pp_fix 2025-12-04T09:23:40.1099866Z * [new branch] whc/sharding -> origin/whc/sharding 2025-12-04T09:23:40.1100048Z * [new branch] whc/sharding2 -> origin/whc/sharding2 2025-12-04T09:23:40.1100173Z * [new branch] whc/uneven -> origin/whc/uneven 2025-12-04T09:23:40.1100454Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-12-04T09:23:40.1101088Z * [new branch] win_warnings -> origin/win_warnings 2025-12-04T09:23:40.1101280Z * [new branch] windows_libtorch_free -> origin/windows_libtorch_free 2025-12-04T09:23:40.1101422Z * [new branch] xmfan-war -> origin/xmfan-war 2025-12-04T09:23:40.1101789Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-12-04T09:23:40.1101947Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-12-04T09:23:40.1102402Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-12-04T09:23:40.1103049Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-12-04T09:23:40.1108048Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-12-04T09:23:40.1108225Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-12-04T09:23:40.1108377Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-12-04T09:23:40.1108521Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-12-04T09:23:40.1108679Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-12-04T09:23:40.1108830Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-12-04T09:23:40.1108990Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-12-04T09:23:40.1109305Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-12-04T09:23:40.1111145Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-12-04T09:23:40.1111479Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-12-04T09:23:40.1111802Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-12-04T09:23:40.1113630Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-12-04T09:23:40.1113982Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-12-04T09:23:40.1114258Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-12-04T09:23:40.1114421Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-12-04T09:23:40.1117783Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-12-04T09:23:40.1118006Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-12-04T09:23:40.1118214Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-12-04T09:23:40.1118516Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-12-04T09:23:40.1119284Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-12-04T09:23:40.1119451Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-12-04T09:23:40.1119605Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-12-04T09:23:40.1120127Z * [new branch] xmfan/test -> origin/xmfan/test 2025-12-04T09:23:40.1121624Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-12-04T09:23:40.1121933Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-12-04T09:23:40.1122476Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-12-04T09:23:40.1124347Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-12-04T09:23:40.1124682Z * [new branch] yiming/run_with_start_end_rng_hop -> origin/yiming/run_with_start_end_rng_hop 2025-12-04T09:23:40.1127430Z * [new branch] yolo-llama3 -> origin/yolo-llama3 2025-12-04T09:23:40.1128029Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-12-04T09:23:40.1128415Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-12-04T09:23:40.1128603Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-12-04T09:23:40.1128860Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-12-04T09:23:40.1131856Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-12-04T09:23:40.1132042Z * [new branch] zb2p -> origin/zb2p 2025-12-04T09:23:40.1132225Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-12-04T09:23:40.1137382Z * [new branch] zhxchen17/ci/vllm_lora_oom -> origin/zhxchen17/ci/vllm_lora_oom 2025-12-04T09:23:40.1137623Z * [new branch] zhxchen17/ci/vllm_multimodal_oom -> origin/zhxchen17/ci/vllm_multimodal_oom 2025-12-04T09:23:40.1137812Z * [new branch] zhxchen17/ci/vllm_pin -> origin/zhxchen17/ci/vllm_pin 2025-12-04T09:23:40.1138062Z * [new branch] zhxchen17/dynamo/unsafe_drop_all_guards -> origin/zhxchen17/dynamo/unsafe_drop_all_guards 2025-12-04T09:23:40.1138256Z * [new branch] zhxchen17/export/call_override -> origin/zhxchen17/export/call_override 2025-12-04T09:23:40.1138444Z * [new branch] zhxchen17/export/codemod1 -> origin/zhxchen17/export/codemod1 2025-12-04T09:23:40.1138640Z * [new branch] zhxchen17/export/ctx_return -> origin/zhxchen17/export/ctx_return 2025-12-04T09:23:40.1139082Z * [new branch] zhxchen17/export/disable_side_effect_warn -> origin/zhxchen17/export/disable_side_effect_warn 2025-12-04T09:23:40.1142458Z * [new branch] zhxchen17/export/pytree_check -> origin/zhxchen17/export/pytree_check 2025-12-04T09:23:40.1142841Z * [new branch] zhxchen17/precompile/aoti -> origin/zhxchen17/precompile/aoti 2025-12-04T09:23:40.1143071Z * [new branch] zhxchen17/precompile/globals -> origin/zhxchen17/precompile/globals 2025-12-04T09:23:40.1143296Z * [new branch] zhxchen17/precompile/inductor_guards -> origin/zhxchen17/precompile/inductor_guards 2025-12-04T09:23:40.1143467Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-12-04T09:23:40.1143806Z * [new branch] zhxchen17/torch_export_api_update -> origin/zhxchen17/torch_export_api_update 2025-12-04T09:23:40.1148593Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-12-04T09:23:40.1148948Z * [new branch] zxiiro/build-times -> origin/zxiiro/build-times 2025-12-04T09:23:40.1149221Z * [new branch] zxiiro/c7i.2xlarge -> origin/zxiiro/c7i.2xlarge 2025-12-04T09:23:40.1149414Z * [new branch] zxiiro/c7i.2xlarge.h100 -> origin/zxiiro/c7i.2xlarge.h100 2025-12-04T09:23:40.1149633Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-12-04T09:23:40.1149924Z * [new branch] zxiiro/risc64 -> origin/zxiiro/risc64 2025-12-04T09:23:40.1150621Z * [new branch] zxiiro/test-multicloud-arc -> origin/zxiiro/test-multicloud-arc 2025-12-04T09:23:40.1151097Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-12-04T09:23:40.1151258Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-12-04T09:23:40.1151795Z * [new tag] ciflow/b200/115316 -> ciflow/b200/115316 2025-12-04T09:23:40.1152160Z * [new tag] ciflow/b200/160685 -> ciflow/b200/160685 2025-12-04T09:23:40.1152714Z * [new tag] ciflow/b200/161607 -> ciflow/b200/161607 2025-12-04T09:23:40.1153067Z * [new tag] ciflow/b200/161938 -> ciflow/b200/161938 2025-12-04T09:23:40.1153807Z * [new tag] ciflow/b200/167207 -> ciflow/b200/167207 2025-12-04T09:23:40.1154331Z * [new tag] ciflow/b200/167989 -> ciflow/b200/167989 2025-12-04T09:23:40.1154775Z * [new tag] ciflow/b200/168096 -> ciflow/b200/168096 2025-12-04T09:23:40.1155112Z * [new tag] ciflow/b200/168175 -> ciflow/b200/168175 2025-12-04T09:23:40.1155830Z * [new tag] ciflow/b200/168195 -> ciflow/b200/168195 2025-12-04T09:23:40.1156136Z * [new tag] ciflow/b200/169200 -> ciflow/b200/169200 2025-12-04T09:23:40.1156700Z * [new tag] ciflow/b200/169216 -> ciflow/b200/169216 2025-12-04T09:23:40.1157857Z * [new tag] ciflow/b200/169380 -> ciflow/b200/169380 2025-12-04T09:23:40.1158229Z * [new tag] ciflow/b200/169412 -> ciflow/b200/169412 2025-12-04T09:23:40.1158984Z * [new tag] ciflow/b200/169470 -> ciflow/b200/169470 2025-12-04T09:23:40.1159165Z * [new tag] ciflow/b200/169471 -> ciflow/b200/169471 2025-12-04T09:23:40.1160067Z * [new tag] ciflow/b200/169472 -> ciflow/b200/169472 2025-12-04T09:23:40.1160544Z * [new tag] ciflow/b200/169514 -> ciflow/b200/169514 2025-12-04T09:23:40.1160964Z * [new tag] ciflow/b200/169517 -> ciflow/b200/169517 2025-12-04T09:23:40.1162064Z * [new tag] ciflow/binaries/165922 -> ciflow/binaries/165922 2025-12-04T09:23:40.1162358Z * [new tag] ciflow/binaries/169510 -> ciflow/binaries/169510 2025-12-04T09:23:40.1164412Z * [new tag] ciflow/binaries_wheel/157994 -> ciflow/binaries_wheel/157994 2025-12-04T09:23:40.1164590Z * [new tag] ciflow/binaries_wheel/166829 -> ciflow/binaries_wheel/166829 2025-12-04T09:23:40.1164749Z * [new tag] ciflow/binaries_wheel/167972 -> ciflow/binaries_wheel/167972 2025-12-04T09:23:40.1164975Z * [new tag] ciflow/binaries_wheel/167981 -> ciflow/binaries_wheel/167981 2025-12-04T09:23:40.1169181Z * [new tag] ciflow/dynamo/167695 -> ciflow/dynamo/167695 2025-12-04T09:23:40.1169304Z * [new tag] ciflow/dynamo/168096 -> ciflow/dynamo/168096 2025-12-04T09:23:40.1169537Z * [new tag] ciflow/dynamo/169525 -> ciflow/dynamo/169525 2025-12-04T09:23:40.1169756Z * [new tag] ciflow/h100-cutlass-backend/161938 -> ciflow/h100-cutlass-backend/161938 2025-12-04T09:23:40.1170007Z * [new tag] ciflow/h100-cutlass-backend/161940 -> ciflow/h100-cutlass-backend/161940 2025-12-04T09:23:40.1170287Z * [new tag] ciflow/h100-distributed/168923 -> ciflow/h100-distributed/168923 2025-12-04T09:23:40.1170511Z * [new tag] ciflow/h100-symm-mem/167552 -> ciflow/h100-symm-mem/167552 2025-12-04T09:23:40.1170657Z * [new tag] ciflow/h100-symm-mem/168129 -> ciflow/h100-symm-mem/168129 2025-12-04T09:23:40.1170878Z * [new tag] ciflow/h100-symm-mem/168917 -> ciflow/h100-symm-mem/168917 2025-12-04T09:23:40.1171523Z * [new tag] ciflow/h100-symm-mem/169156 -> ciflow/h100-symm-mem/169156 2025-12-04T09:23:40.1171695Z * [new tag] ciflow/h100-symm-mem/169200 -> ciflow/h100-symm-mem/169200 2025-12-04T09:23:40.1171872Z * [new tag] ciflow/h100-symm-mem/169216 -> ciflow/h100-symm-mem/169216 2025-12-04T09:23:40.1172010Z * [new tag] ciflow/h100-symm-mem/169338 -> ciflow/h100-symm-mem/169338 2025-12-04T09:23:40.1172165Z * [new tag] ciflow/h100-symm-mem/169355 -> ciflow/h100-symm-mem/169355 2025-12-04T09:23:40.1172506Z * [new tag] ciflow/h100-symm-mem/169543 -> ciflow/h100-symm-mem/169543 2025-12-04T09:23:40.1173046Z * [new tag] ciflow/h100/115316 -> ciflow/h100/115316 2025-12-04T09:23:40.1174076Z * [new tag] ciflow/h100/160685 -> ciflow/h100/160685 2025-12-04T09:23:40.1174205Z * [new tag] ciflow/h100/160729 -> ciflow/h100/160729 2025-12-04T09:23:40.1174597Z * [new tag] ciflow/h100/161607 -> ciflow/h100/161607 2025-12-04T09:23:40.1174959Z * [new tag] ciflow/h100/161938 -> ciflow/h100/161938 2025-12-04T09:23:40.1175513Z * [new tag] ciflow/h100/167207 -> ciflow/h100/167207 2025-12-04T09:23:40.1175816Z * [new tag] ciflow/h100/167989 -> ciflow/h100/167989 2025-12-04T09:23:40.1176536Z * [new tag] ciflow/h100/168096 -> ciflow/h100/168096 2025-12-04T09:23:40.1177195Z * [new tag] ciflow/h100/168175 -> ciflow/h100/168175 2025-12-04T09:23:40.1177815Z * [new tag] ciflow/h100/168195 -> ciflow/h100/168195 2025-12-04T09:23:40.1178091Z * [new tag] ciflow/h100/168980 -> ciflow/h100/168980 2025-12-04T09:23:40.1178216Z * [new tag] ciflow/h100/169200 -> ciflow/h100/169200 2025-12-04T09:23:40.1179595Z * [new tag] ciflow/h100/169216 -> ciflow/h100/169216 2025-12-04T09:23:40.1179722Z * [new tag] ciflow/h100/169380 -> ciflow/h100/169380 2025-12-04T09:23:40.1179834Z * [new tag] ciflow/h100/169412 -> ciflow/h100/169412 2025-12-04T09:23:40.1180049Z * [new tag] ciflow/h100/169470 -> ciflow/h100/169470 2025-12-04T09:23:40.1183619Z * [new tag] ciflow/h100/169471 -> ciflow/h100/169471 2025-12-04T09:23:40.1183777Z * [new tag] ciflow/h100/169472 -> ciflow/h100/169472 2025-12-04T09:23:40.1183906Z * [new tag] ciflow/h100/169514 -> ciflow/h100/169514 2025-12-04T09:23:40.1184067Z * [new tag] ciflow/inductor-cu126/168096 -> ciflow/inductor-cu126/168096 2025-12-04T09:23:40.1184398Z * [new tag] ciflow/inductor-micro-benchmark-cpu-x86/168096 -> ciflow/inductor-micro-benchmark-cpu-x86/168096 2025-12-04T09:23:40.1184600Z * [new tag] ciflow/inductor-micro-benchmark/166165 -> ciflow/inductor-micro-benchmark/166165 2025-12-04T09:23:40.1184830Z * [new tag] ciflow/inductor-micro-benchmark/168096 -> ciflow/inductor-micro-benchmark/168096 2025-12-04T09:23:40.1185067Z * [new tag] ciflow/inductor-perf-compare/168096 -> ciflow/inductor-perf-compare/168096 2025-12-04T09:23:40.1185455Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/168073 -> ciflow/inductor-perf-test-nightly-rocm-mi300/168073 2025-12-04T09:23:40.1185743Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/168096 -> ciflow/inductor-perf-test-nightly-rocm-mi300/168096 2025-12-04T09:23:40.1186234Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/169024 -> ciflow/inductor-perf-test-nightly-rocm-mi300/169024 2025-12-04T09:23:40.1186630Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi355/169024 -> ciflow/inductor-perf-test-nightly-rocm-mi355/169024 2025-12-04T09:23:40.1187393Z * [new tag] ciflow/inductor-perf-test-nightly/168096 -> ciflow/inductor-perf-test-nightly/168096 2025-12-04T09:23:40.1187635Z * [new tag] ciflow/inductor-periodic/168096 -> ciflow/inductor-periodic/168096 2025-12-04T09:23:40.1187803Z * [new tag] ciflow/inductor-periodic/169024 -> ciflow/inductor-periodic/169024 2025-12-04T09:23:40.1188211Z * [new tag] ciflow/inductor-periodic/169425 -> ciflow/inductor-periodic/169425 2025-12-04T09:23:40.1189158Z * [new tag] ciflow/inductor-rocm-mi200/165545 -> ciflow/inductor-rocm-mi200/165545 2025-12-04T09:23:40.1189415Z * [new tag] ciflow/inductor-rocm-mi200/165997 -> ciflow/inductor-rocm-mi200/165997 2025-12-04T09:23:40.1189776Z * [new tag] ciflow/inductor-rocm-mi200/168096 -> ciflow/inductor-rocm-mi200/168096 2025-12-04T09:23:40.1190291Z * [new tag] ciflow/inductor-rocm-mi200/169063 -> ciflow/inductor-rocm-mi200/169063 2025-12-04T09:23:40.1190727Z * [new tag] ciflow/inductor-rocm-mi200/169425 -> ciflow/inductor-rocm-mi200/169425 2025-12-04T09:23:40.1192188Z * [new tag] ciflow/inductor-rocm-mi300/165545 -> ciflow/inductor-rocm-mi300/165545 2025-12-04T09:23:40.1192557Z * [new tag] ciflow/inductor-rocm-mi300/168096 -> ciflow/inductor-rocm-mi300/168096 2025-12-04T09:23:40.1192830Z * [new tag] ciflow/inductor-rocm-mi300/169063 -> ciflow/inductor-rocm-mi300/169063 2025-12-04T09:23:40.1193118Z * [new tag] ciflow/inductor-rocm-mi300/169425 -> ciflow/inductor-rocm-mi300/169425 2025-12-04T09:23:40.1193396Z * [new tag] ciflow/inductor-rocm/162052 -> ciflow/inductor-rocm/162052 2025-12-04T09:23:40.1193790Z * [new tag] ciflow/inductor-rocm/168971 -> ciflow/inductor-rocm/168971 2025-12-04T09:23:40.1194979Z * [new tag] ciflow/inductor-windows/168096 -> ciflow/inductor-windows/168096 2025-12-04T09:23:40.1195142Z * [new tag] ciflow/inductor/144542 -> ciflow/inductor/144542 2025-12-04T09:23:40.1195426Z * [new tag] ciflow/inductor/146506 -> ciflow/inductor/146506 2025-12-04T09:23:40.1195919Z * [new tag] ciflow/inductor/147990 -> ciflow/inductor/147990 2025-12-04T09:23:40.1198572Z * [new tag] ciflow/inductor/148294 -> ciflow/inductor/148294 2025-12-04T09:23:40.1198903Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-12-04T09:23:40.1199048Z * [new tag] ciflow/inductor/157149 -> ciflow/inductor/157149 2025-12-04T09:23:40.1199175Z * [new tag] ciflow/inductor/157994 -> ciflow/inductor/157994 2025-12-04T09:23:40.1199306Z * [new tag] ciflow/inductor/160685 -> ciflow/inductor/160685 2025-12-04T09:23:40.1199460Z * [new tag] ciflow/inductor/160686 -> ciflow/inductor/160686 2025-12-04T09:23:40.1199585Z * [new tag] ciflow/inductor/160687 -> ciflow/inductor/160687 2025-12-04T09:23:40.1200066Z * [new tag] ciflow/inductor/160688 -> ciflow/inductor/160688 2025-12-04T09:23:40.1200415Z * [new tag] ciflow/inductor/160706 -> ciflow/inductor/160706 2025-12-04T09:23:40.1201413Z * [new tag] ciflow/inductor/160729 -> ciflow/inductor/160729 2025-12-04T09:23:40.1201566Z * [new tag] ciflow/inductor/161938 -> ciflow/inductor/161938 2025-12-04T09:23:40.1204650Z * [new tag] ciflow/inductor/161939 -> ciflow/inductor/161939 2025-12-04T09:23:40.1204822Z * [new tag] ciflow/inductor/161940 -> ciflow/inductor/161940 2025-12-04T09:23:40.1204964Z * [new tag] ciflow/inductor/162052 -> ciflow/inductor/162052 2025-12-04T09:23:40.1205126Z * [new tag] ciflow/inductor/162275 -> ciflow/inductor/162275 2025-12-04T09:23:40.1205261Z * [new tag] ciflow/inductor/162795 -> ciflow/inductor/162795 2025-12-04T09:23:40.1205651Z * [new tag] ciflow/inductor/163245 -> ciflow/inductor/163245 2025-12-04T09:23:40.1205860Z * [new tag] ciflow/inductor/163335 -> ciflow/inductor/163335 2025-12-04T09:23:40.1206095Z * [new tag] ciflow/inductor/163503 -> ciflow/inductor/163503 2025-12-04T09:23:40.1206237Z * [new tag] ciflow/inductor/163942 -> ciflow/inductor/163942 2025-12-04T09:23:40.1206683Z * [new tag] ciflow/inductor/165270 -> ciflow/inductor/165270 2025-12-04T09:23:40.1210504Z * [new tag] ciflow/inductor/165274 -> ciflow/inductor/165274 2025-12-04T09:23:40.1210817Z * [new tag] ciflow/inductor/165322 -> ciflow/inductor/165322 2025-12-04T09:23:40.1211235Z * [new tag] ciflow/inductor/165597 -> ciflow/inductor/165597 2025-12-04T09:23:40.1211386Z * [new tag] ciflow/inductor/166063 -> ciflow/inductor/166063 2025-12-04T09:23:40.1211524Z * [new tag] ciflow/inductor/166075 -> ciflow/inductor/166075 2025-12-04T09:23:40.1211865Z * [new tag] ciflow/inductor/166165 -> ciflow/inductor/166165 2025-12-04T09:23:40.1212403Z * [new tag] ciflow/inductor/166254 -> ciflow/inductor/166254 2025-12-04T09:23:40.1212574Z * [new tag] ciflow/inductor/166483 -> ciflow/inductor/166483 2025-12-04T09:23:40.1212705Z * [new tag] ciflow/inductor/166494 -> ciflow/inductor/166494 2025-12-04T09:23:40.1212834Z * [new tag] ciflow/inductor/166545 -> ciflow/inductor/166545 2025-12-04T09:23:40.1212978Z * [new tag] ciflow/inductor/166788 -> ciflow/inductor/166788 2025-12-04T09:23:40.1213119Z * [new tag] ciflow/inductor/166846 -> ciflow/inductor/166846 2025-12-04T09:23:40.1213294Z * [new tag] ciflow/inductor/167300 -> ciflow/inductor/167300 2025-12-04T09:23:40.1213746Z * [new tag] ciflow/inductor/167407 -> ciflow/inductor/167407 2025-12-04T09:23:40.1218279Z * [new tag] ciflow/inductor/167536 -> ciflow/inductor/167536 2025-12-04T09:23:40.1218453Z * [new tag] ciflow/inductor/167552 -> ciflow/inductor/167552 2025-12-04T09:23:40.1218592Z * [new tag] ciflow/inductor/167555 -> ciflow/inductor/167555 2025-12-04T09:23:40.1218718Z * [new tag] ciflow/inductor/167583 -> ciflow/inductor/167583 2025-12-04T09:23:40.1218845Z * [new tag] ciflow/inductor/167599 -> ciflow/inductor/167599 2025-12-04T09:23:40.1218977Z * [new tag] ciflow/inductor/167647 -> ciflow/inductor/167647 2025-12-04T09:23:40.1219114Z * [new tag] ciflow/inductor/167677 -> ciflow/inductor/167677 2025-12-04T09:23:40.1219241Z * [new tag] ciflow/inductor/167680 -> ciflow/inductor/167680 2025-12-04T09:23:40.1219373Z * [new tag] ciflow/inductor/167695 -> ciflow/inductor/167695 2025-12-04T09:23:40.1219500Z * [new tag] ciflow/inductor/167742 -> ciflow/inductor/167742 2025-12-04T09:23:40.1219635Z * [new tag] ciflow/inductor/167768 -> ciflow/inductor/167768 2025-12-04T09:23:40.1219778Z * [new tag] ciflow/inductor/167773 -> ciflow/inductor/167773 2025-12-04T09:23:40.1224339Z * [new tag] ciflow/inductor/167781 -> ciflow/inductor/167781 2025-12-04T09:23:40.1224996Z * [new tag] ciflow/inductor/167880 -> ciflow/inductor/167880 2025-12-04T09:23:40.1225182Z * [new tag] ciflow/inductor/167887 -> ciflow/inductor/167887 2025-12-04T09:23:40.1225332Z * [new tag] ciflow/inductor/167972 -> ciflow/inductor/167972 2025-12-04T09:23:40.1225469Z * [new tag] ciflow/inductor/167989 -> ciflow/inductor/167989 2025-12-04T09:23:40.1225601Z * [new tag] ciflow/inductor/168002 -> ciflow/inductor/168002 2025-12-04T09:23:40.1225729Z * [new tag] ciflow/inductor/168050 -> ciflow/inductor/168050 2025-12-04T09:23:40.1225882Z * [new tag] ciflow/inductor/168051 -> ciflow/inductor/168051 2025-12-04T09:23:40.1226007Z * [new tag] ciflow/inductor/168052 -> ciflow/inductor/168052 2025-12-04T09:23:40.1226137Z * [new tag] ciflow/inductor/168073 -> ciflow/inductor/168073 2025-12-04T09:23:40.1226267Z * [new tag] ciflow/inductor/168096 -> ciflow/inductor/168096 2025-12-04T09:23:40.1226391Z * [new tag] ciflow/inductor/168114 -> ciflow/inductor/168114 2025-12-04T09:23:40.1226682Z * [new tag] ciflow/inductor/168115 -> ciflow/inductor/168115 2025-12-04T09:23:40.1227054Z * [new tag] ciflow/inductor/168127 -> ciflow/inductor/168127 2025-12-04T09:23:40.1227205Z * [new tag] ciflow/inductor/168129 -> ciflow/inductor/168129 2025-12-04T09:23:40.1229101Z * [new tag] ciflow/inductor/168157 -> ciflow/inductor/168157 2025-12-04T09:23:40.1229427Z * [new tag] ciflow/inductor/168175 -> ciflow/inductor/168175 2025-12-04T09:23:40.1229584Z * [new tag] ciflow/inductor/168185 -> ciflow/inductor/168185 2025-12-04T09:23:40.1229716Z * [new tag] ciflow/inductor/168195 -> ciflow/inductor/168195 2025-12-04T09:23:40.1229885Z * [new tag] ciflow/inductor/168209 -> ciflow/inductor/168209 2025-12-04T09:23:40.1230239Z * [new tag] ciflow/inductor/168266 -> ciflow/inductor/168266 2025-12-04T09:23:40.1230778Z * [new tag] ciflow/inductor/168316 -> ciflow/inductor/168316 2025-12-04T09:23:40.1231316Z * [new tag] ciflow/inductor/168326 -> ciflow/inductor/168326 2025-12-04T09:23:40.1231732Z * [new tag] ciflow/inductor/168368 -> ciflow/inductor/168368 2025-12-04T09:23:40.1235361Z * [new tag] ciflow/inductor/168894 -> ciflow/inductor/168894 2025-12-04T09:23:40.1235557Z * [new tag] ciflow/inductor/168934 -> ciflow/inductor/168934 2025-12-04T09:23:40.1235703Z * [new tag] ciflow/inductor/168939 -> ciflow/inductor/168939 2025-12-04T09:23:40.1235836Z * [new tag] ciflow/inductor/168946 -> ciflow/inductor/168946 2025-12-04T09:23:40.1235975Z * [new tag] ciflow/inductor/168950 -> ciflow/inductor/168950 2025-12-04T09:23:40.1236107Z * [new tag] ciflow/inductor/168951 -> ciflow/inductor/168951 2025-12-04T09:23:40.1236235Z * [new tag] ciflow/inductor/168952 -> ciflow/inductor/168952 2025-12-04T09:23:40.1236388Z * [new tag] ciflow/inductor/168955 -> ciflow/inductor/168955 2025-12-04T09:23:40.1236522Z * [new tag] ciflow/inductor/168971 -> ciflow/inductor/168971 2025-12-04T09:23:40.1237212Z * [new tag] ciflow/inductor/168979 -> ciflow/inductor/168979 2025-12-04T09:23:40.1237544Z * [new tag] ciflow/inductor/168980 -> ciflow/inductor/168980 2025-12-04T09:23:40.1238521Z * [new tag] ciflow/inductor/168983 -> ciflow/inductor/168983 2025-12-04T09:23:40.1238846Z * [new tag] ciflow/inductor/169006 -> ciflow/inductor/169006 2025-12-04T09:23:40.1239148Z * [new tag] ciflow/inductor/169023 -> ciflow/inductor/169023 2025-12-04T09:23:40.1239632Z * [new tag] ciflow/inductor/169024 -> ciflow/inductor/169024 2025-12-04T09:23:40.1240233Z * [new tag] ciflow/inductor/169025 -> ciflow/inductor/169025 2025-12-04T09:23:40.1240871Z * [new tag] ciflow/inductor/169066 -> ciflow/inductor/169066 2025-12-04T09:23:40.1241194Z * [new tag] ciflow/inductor/169091 -> ciflow/inductor/169091 2025-12-04T09:23:40.1241743Z * [new tag] ciflow/inductor/169102 -> ciflow/inductor/169102 2025-12-04T09:23:40.1242147Z * [new tag] ciflow/inductor/169103 -> ciflow/inductor/169103 2025-12-04T09:23:40.1242681Z * [new tag] ciflow/inductor/169121 -> ciflow/inductor/169121 2025-12-04T09:23:40.1243134Z * [new tag] ciflow/inductor/169134 -> ciflow/inductor/169134 2025-12-04T09:23:40.1246364Z * [new tag] ciflow/inductor/169135 -> ciflow/inductor/169135 2025-12-04T09:23:40.1246540Z * [new tag] ciflow/inductor/169141 -> ciflow/inductor/169141 2025-12-04T09:23:40.1246683Z * [new tag] ciflow/inductor/169151 -> ciflow/inductor/169151 2025-12-04T09:23:40.1246990Z * [new tag] ciflow/inductor/169161 -> ciflow/inductor/169161 2025-12-04T09:23:40.1247316Z * [new tag] ciflow/inductor/169167 -> ciflow/inductor/169167 2025-12-04T09:23:40.1247465Z * [new tag] ciflow/inductor/169177 -> ciflow/inductor/169177 2025-12-04T09:23:40.1247863Z * [new tag] ciflow/inductor/169185 -> ciflow/inductor/169185 2025-12-04T09:23:40.1253013Z * [new tag] ciflow/inductor/169196 -> ciflow/inductor/169196 2025-12-04T09:23:40.1256551Z * [new tag] ciflow/inductor/169200 -> ciflow/inductor/169200 2025-12-04T09:23:40.1256718Z * [new tag] ciflow/inductor/169204 -> ciflow/inductor/169204 2025-12-04T09:23:40.1260022Z * [new tag] ciflow/inductor/169216 -> ciflow/inductor/169216 2025-12-04T09:23:40.1260156Z * [new tag] ciflow/inductor/169219 -> ciflow/inductor/169219 2025-12-04T09:23:40.1260399Z * [new tag] ciflow/inductor/169220 -> ciflow/inductor/169220 2025-12-04T09:23:40.1264554Z * [new tag] ciflow/inductor/169230 -> ciflow/inductor/169230 2025-12-04T09:23:40.1264724Z * [new tag] ciflow/inductor/169242 -> ciflow/inductor/169242 2025-12-04T09:23:40.1264875Z * [new tag] ciflow/inductor/169245 -> ciflow/inductor/169245 2025-12-04T09:23:40.1265015Z * [new tag] ciflow/inductor/169260 -> ciflow/inductor/169260 2025-12-04T09:23:40.1265147Z * [new tag] ciflow/inductor/169282 -> ciflow/inductor/169282 2025-12-04T09:23:40.1265331Z * [new tag] ciflow/inductor/169286 -> ciflow/inductor/169286 2025-12-04T09:23:40.1265459Z * [new tag] ciflow/inductor/169299 -> ciflow/inductor/169299 2025-12-04T09:23:40.1265622Z * [new tag] ciflow/inductor/169304 -> ciflow/inductor/169304 2025-12-04T09:23:40.1265752Z * [new tag] ciflow/inductor/169305 -> ciflow/inductor/169305 2025-12-04T09:23:40.1265887Z * [new tag] ciflow/inductor/169308 -> ciflow/inductor/169308 2025-12-04T09:23:40.1266014Z * [new tag] ciflow/inductor/169319 -> ciflow/inductor/169319 2025-12-04T09:23:40.1266141Z * [new tag] ciflow/inductor/169326 -> ciflow/inductor/169326 2025-12-04T09:23:40.1266279Z * [new tag] ciflow/inductor/169332 -> ciflow/inductor/169332 2025-12-04T09:23:40.1266405Z * [new tag] ciflow/inductor/169333 -> ciflow/inductor/169333 2025-12-04T09:23:40.1266527Z * [new tag] ciflow/inductor/169336 -> ciflow/inductor/169336 2025-12-04T09:23:40.1266657Z * [new tag] ciflow/inductor/169340 -> ciflow/inductor/169340 2025-12-04T09:23:40.1266781Z * [new tag] ciflow/inductor/169341 -> ciflow/inductor/169341 2025-12-04T09:23:40.1266913Z * [new tag] ciflow/inductor/169343 -> ciflow/inductor/169343 2025-12-04T09:23:40.1267042Z * [new tag] ciflow/inductor/169346 -> ciflow/inductor/169346 2025-12-04T09:23:40.1267167Z * [new tag] ciflow/inductor/169348 -> ciflow/inductor/169348 2025-12-04T09:23:40.1267298Z * [new tag] ciflow/inductor/169350 -> ciflow/inductor/169350 2025-12-04T09:23:40.1267430Z * [new tag] ciflow/inductor/169355 -> ciflow/inductor/169355 2025-12-04T09:23:40.1267562Z * [new tag] ciflow/inductor/169370 -> ciflow/inductor/169370 2025-12-04T09:23:40.1267689Z * [new tag] ciflow/inductor/169375 -> ciflow/inductor/169375 2025-12-04T09:23:40.1267814Z * [new tag] ciflow/inductor/169389 -> ciflow/inductor/169389 2025-12-04T09:23:40.1267945Z * [new tag] ciflow/inductor/169391 -> ciflow/inductor/169391 2025-12-04T09:23:40.1268071Z * [new tag] ciflow/inductor/169393 -> ciflow/inductor/169393 2025-12-04T09:23:40.1268405Z * [new tag] ciflow/inductor/169399 -> ciflow/inductor/169399 2025-12-04T09:23:40.1268531Z * [new tag] ciflow/inductor/169400 -> ciflow/inductor/169400 2025-12-04T09:23:40.1268655Z * [new tag] ciflow/inductor/169415 -> ciflow/inductor/169415 2025-12-04T09:23:40.1268834Z * [new tag] ciflow/inductor/169417 -> ciflow/inductor/169417 2025-12-04T09:23:40.1268964Z * [new tag] ciflow/inductor/169418 -> ciflow/inductor/169418 2025-12-04T09:23:40.1269411Z * [new tag] ciflow/inductor/169430 -> ciflow/inductor/169430 2025-12-04T09:23:40.1269555Z * [new tag] ciflow/inductor/169432 -> ciflow/inductor/169432 2025-12-04T09:23:40.1270134Z * [new tag] ciflow/inductor/169436 -> ciflow/inductor/169436 2025-12-04T09:23:40.1270481Z * [new tag] ciflow/inductor/169437 -> ciflow/inductor/169437 2025-12-04T09:23:40.1271508Z * [new tag] ciflow/inductor/169438 -> ciflow/inductor/169438 2025-12-04T09:23:40.1271784Z * [new tag] ciflow/inductor/169441 -> ciflow/inductor/169441 2025-12-04T09:23:40.1272111Z * [new tag] ciflow/inductor/169446 -> ciflow/inductor/169446 2025-12-04T09:23:40.1272913Z * [new tag] ciflow/inductor/169447 -> ciflow/inductor/169447 2025-12-04T09:23:40.1273236Z * [new tag] ciflow/inductor/169452 -> ciflow/inductor/169452 2025-12-04T09:23:40.1273839Z * [new tag] ciflow/inductor/169455 -> ciflow/inductor/169455 2025-12-04T09:23:40.1274232Z * [new tag] ciflow/inductor/169459 -> ciflow/inductor/169459 2025-12-04T09:23:40.1275220Z * [new tag] ciflow/inductor/169463 -> ciflow/inductor/169463 2025-12-04T09:23:40.1275489Z * [new tag] ciflow/inductor/169476 -> ciflow/inductor/169476 2025-12-04T09:23:40.1276355Z * [new tag] ciflow/inductor/169485 -> ciflow/inductor/169485 2025-12-04T09:23:40.1276914Z * [new tag] ciflow/inductor/169493 -> ciflow/inductor/169493 2025-12-04T09:23:40.1277205Z * [new tag] ciflow/inductor/169496 -> ciflow/inductor/169496 2025-12-04T09:23:40.1277545Z * [new tag] ciflow/inductor/169497 -> ciflow/inductor/169497 2025-12-04T09:23:40.1278091Z * [new tag] ciflow/inductor/169503 -> ciflow/inductor/169503 2025-12-04T09:23:40.1278592Z * [new tag] ciflow/inductor/169504 -> ciflow/inductor/169504 2025-12-04T09:23:40.1279628Z * [new tag] ciflow/inductor/169505 -> ciflow/inductor/169505 2025-12-04T09:23:40.1280147Z * [new tag] ciflow/inductor/169508 -> ciflow/inductor/169508 2025-12-04T09:23:40.1280933Z * [new tag] ciflow/inductor/169509 -> ciflow/inductor/169509 2025-12-04T09:23:40.1281493Z * [new tag] ciflow/inductor/169513 -> ciflow/inductor/169513 2025-12-04T09:23:40.1281717Z * [new tag] ciflow/inductor/169514 -> ciflow/inductor/169514 2025-12-04T09:23:40.1282266Z * [new tag] ciflow/inductor/169515 -> ciflow/inductor/169515 2025-12-04T09:23:40.1282712Z * [new tag] ciflow/inductor/169517 -> ciflow/inductor/169517 2025-12-04T09:23:40.1283183Z * [new tag] ciflow/inductor/169519 -> ciflow/inductor/169519 2025-12-04T09:23:40.1287233Z * [new tag] ciflow/inductor/169520 -> ciflow/inductor/169520 2025-12-04T09:23:40.1287422Z * [new tag] ciflow/inductor/169521 -> ciflow/inductor/169521 2025-12-04T09:23:40.1287563Z * [new tag] ciflow/inductor/169524 -> ciflow/inductor/169524 2025-12-04T09:23:40.1287689Z * [new tag] ciflow/inductor/169527 -> ciflow/inductor/169527 2025-12-04T09:23:40.1288001Z * [new tag] ciflow/inductor/169528 -> ciflow/inductor/169528 2025-12-04T09:23:40.1288131Z * [new tag] ciflow/inductor/169532 -> ciflow/inductor/169532 2025-12-04T09:23:40.1288263Z * [new tag] ciflow/inductor/169535 -> ciflow/inductor/169535 2025-12-04T09:23:40.1288395Z * [new tag] ciflow/inductor/169536 -> ciflow/inductor/169536 2025-12-04T09:23:40.1288596Z * [new tag] ciflow/inductor/169547 -> ciflow/inductor/169547 2025-12-04T09:23:40.1288722Z * [new tag] ciflow/inductor/169548 -> ciflow/inductor/169548 2025-12-04T09:23:40.1288859Z * [new tag] ciflow/inductor/169549 -> ciflow/inductor/169549 2025-12-04T09:23:40.1289176Z * [new tag] ciflow/inductor/169551 -> ciflow/inductor/169551 2025-12-04T09:23:40.1289671Z * [new tag] ciflow/inductor/169552 -> ciflow/inductor/169552 2025-12-04T09:23:40.1290092Z * [new tag] ciflow/inductor/169553 -> ciflow/inductor/169553 2025-12-04T09:23:40.1290818Z * [new tag] ciflow/inductor/169557 -> ciflow/inductor/169557 2025-12-04T09:23:40.1293942Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-12-04T09:23:40.1294278Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-12-04T09:23:40.1294453Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-12-04T09:23:40.1294716Z * [new tag] ciflow/linux-aarch64/157994 -> ciflow/linux-aarch64/157994 2025-12-04T09:23:40.1294877Z * [new tag] ciflow/linux-aarch64/166075 -> ciflow/linux-aarch64/166075 2025-12-04T09:23:40.1295181Z * [new tag] ciflow/linux-aarch64/166876 -> ciflow/linux-aarch64/166876 2025-12-04T09:23:40.1295345Z * [new tag] ciflow/linux-aarch64/167981 -> ciflow/linux-aarch64/167981 2025-12-04T09:23:40.1300066Z * [new tag] ciflow/mps/166254 -> ciflow/mps/166254 2025-12-04T09:23:40.1300373Z * [new tag] ciflow/mps/169017 -> ciflow/mps/169017 2025-12-04T09:23:40.1300524Z * [new tag] ciflow/mps/169372 -> ciflow/mps/169372 2025-12-04T09:23:40.1300641Z * [new tag] ciflow/mps/169478 -> ciflow/mps/169478 2025-12-04T09:23:40.1300802Z * [new tag] ciflow/op-benchmark/157994 -> ciflow/op-benchmark/157994 2025-12-04T09:23:40.1300949Z * [new tag] ciflow/op-benchmark/166075 -> ciflow/op-benchmark/166075 2025-12-04T09:23:40.1301099Z * [new tag] ciflow/op-benchmark/169544 -> ciflow/op-benchmark/169544 2025-12-04T09:23:40.1301436Z * [new tag] ciflow/periodic-rocm-mi200/165997 -> ciflow/periodic-rocm-mi200/165997 2025-12-04T09:23:40.1301655Z * [new tag] ciflow/periodic-rocm-mi200/166517 -> ciflow/periodic-rocm-mi200/166517 2025-12-04T09:23:40.1302303Z * [new tag] ciflow/periodic-rocm-mi200/169063 -> ciflow/periodic-rocm-mi200/169063 2025-12-04T09:23:40.1302530Z * [new tag] ciflow/periodic-rocm-mi200/169425 -> ciflow/periodic-rocm-mi200/169425 2025-12-04T09:23:40.1302710Z * [new tag] ciflow/periodic-rocm-mi300/166517 -> ciflow/periodic-rocm-mi300/166517 2025-12-04T09:23:40.1302894Z * [new tag] ciflow/periodic-rocm-mi300/169063 -> ciflow/periodic-rocm-mi300/169063 2025-12-04T09:23:40.1303063Z * [new tag] ciflow/periodic-rocm-mi300/169425 -> ciflow/periodic-rocm-mi300/169425 2025-12-04T09:23:40.1303222Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-12-04T09:23:40.1303362Z * [new tag] ciflow/periodic/167207 -> ciflow/periodic/167207 2025-12-04T09:23:40.1303671Z * [new tag] ciflow/periodic/167978 -> ciflow/periodic/167978 2025-12-04T09:23:40.1304325Z * [new tag] ciflow/periodic/168096 -> ciflow/periodic/168096 2025-12-04T09:23:40.1304702Z * [new tag] ciflow/periodic/169286 -> ciflow/periodic/169286 2025-12-04T09:23:40.1304854Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-12-04T09:23:40.1305164Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-12-04T09:23:40.1305362Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-12-04T09:23:40.1307639Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-12-04T09:23:40.1308138Z * [new tag] ciflow/periodic/7c648509a7470ace9fb2bae960dd4790f7e943e9 -> ciflow/periodic/7c648509a7470ace9fb2bae960dd4790f7e943e9 2025-12-04T09:23:40.1308432Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-12-04T09:23:40.1308611Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-12-04T09:23:40.1309238Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-12-04T09:23:40.1309672Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-12-04T09:23:40.1310508Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-12-04T09:23:40.1310984Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-12-04T09:23:40.1315028Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-12-04T09:23:40.1315351Z * [new tag] ciflow/pull/167207 -> ciflow/pull/167207 2025-12-04T09:23:40.1315585Z * [new tag] ciflow/quantization-periodic/169207 -> ciflow/quantization-periodic/169207 2025-12-04T09:23:40.1315737Z * [new tag] ciflow/rocm-mi200/165545 -> ciflow/rocm-mi200/165545 2025-12-04T09:23:40.1315966Z * [new tag] ciflow/rocm-mi200/165997 -> ciflow/rocm-mi200/165997 2025-12-04T09:23:40.1316140Z * [new tag] ciflow/rocm-mi200/168096 -> ciflow/rocm-mi200/168096 2025-12-04T09:23:40.1316370Z * [new tag] ciflow/rocm-mi200/168275 -> ciflow/rocm-mi200/168275 2025-12-04T09:23:40.1317041Z * [new tag] ciflow/rocm-mi200/169063 -> ciflow/rocm-mi200/169063 2025-12-04T09:23:40.1317229Z * [new tag] ciflow/rocm-mi200/169356 -> ciflow/rocm-mi200/169356 2025-12-04T09:23:40.1317368Z * [new tag] ciflow/rocm-mi200/169425 -> ciflow/rocm-mi200/169425 2025-12-04T09:23:40.1317512Z * [new tag] ciflow/rocm-mi300/165545 -> ciflow/rocm-mi300/165545 2025-12-04T09:23:40.1318807Z * [new tag] ciflow/rocm-mi300/167157 -> ciflow/rocm-mi300/167157 2025-12-04T09:23:40.1319104Z * [new tag] ciflow/rocm-mi300/168096 -> ciflow/rocm-mi300/168096 2025-12-04T09:23:40.1319257Z * [new tag] ciflow/rocm-mi300/169063 -> ciflow/rocm-mi300/169063 2025-12-04T09:23:40.1319502Z * [new tag] ciflow/rocm-mi300/169425 -> ciflow/rocm-mi300/169425 2025-12-04T09:23:40.1319947Z * [new tag] ciflow/rocm-mi355/167157 -> ciflow/rocm-mi355/167157 2025-12-04T09:23:40.1320465Z * [new tag] ciflow/rocm-mi355/168275 -> ciflow/rocm-mi355/168275 2025-12-04T09:23:40.1320658Z * [new tag] ciflow/rocm-mi355/169425 -> ciflow/rocm-mi355/169425 2025-12-04T09:23:40.1321724Z * [new tag] ciflow/rocm-navi31/168275 -> ciflow/rocm-navi31/168275 2025-12-04T09:23:40.1321873Z * [new tag] ciflow/rocm-navi31/169425 -> ciflow/rocm-navi31/169425 2025-12-04T09:23:40.1322380Z * [new tag] ciflow/rocm/115316 -> ciflow/rocm/115316 2025-12-04T09:23:40.1322833Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-12-04T09:23:40.1323301Z * [new tag] ciflow/rocm/160685 -> ciflow/rocm/160685 2025-12-04T09:23:40.1323862Z * [new tag] ciflow/rocm/161607 -> ciflow/rocm/161607 2025-12-04T09:23:40.1325379Z * [new tag] ciflow/rocm/162052 -> ciflow/rocm/162052 2025-12-04T09:23:40.1325613Z * [new tag] ciflow/rocm/165997 -> ciflow/rocm/165997 2025-12-04T09:23:40.1325842Z * [new tag] ciflow/rocm/166165 -> ciflow/rocm/166165 2025-12-04T09:23:40.1326104Z * [new tag] ciflow/rocm/166517 -> ciflow/rocm/166517 2025-12-04T09:23:40.1326237Z * [new tag] ciflow/rocm/167207 -> ciflow/rocm/167207 2025-12-04T09:23:40.1326610Z * [new tag] ciflow/rocm/167536 -> ciflow/rocm/167536 2025-12-04T09:23:40.1327071Z * [new tag] ciflow/rocm/167781 -> ciflow/rocm/167781 2025-12-04T09:23:40.1328334Z * [new tag] ciflow/rocm/167989 -> ciflow/rocm/167989 2025-12-04T09:23:40.1328688Z * [new tag] ciflow/rocm/168073 -> ciflow/rocm/168073 2025-12-04T09:23:40.1328869Z * [new tag] ciflow/rocm/168195 -> ciflow/rocm/168195 2025-12-04T09:23:40.1330836Z * [new tag] ciflow/rocm/168939 -> ciflow/rocm/168939 2025-12-04T09:23:40.1331157Z * [new tag] ciflow/rocm/168971 -> ciflow/rocm/168971 2025-12-04T09:23:40.1331314Z * [new tag] ciflow/rocm/169024 -> ciflow/rocm/169024 2025-12-04T09:23:40.1331533Z * [new tag] ciflow/rocm/169200 -> ciflow/rocm/169200 2025-12-04T09:23:40.1331668Z * [new tag] ciflow/rocm/169216 -> ciflow/rocm/169216 2025-12-04T09:23:40.1333227Z * [new tag] ciflow/rocm/169312 -> ciflow/rocm/169312 2025-12-04T09:23:40.1333553Z * [new tag] ciflow/rocm/169380 -> ciflow/rocm/169380 2025-12-04T09:23:40.1333698Z * [new tag] ciflow/rocm/169427 -> ciflow/rocm/169427 2025-12-04T09:23:40.1334049Z * [new tag] ciflow/rocm/169455 -> ciflow/rocm/169455 2025-12-04T09:23:40.1334530Z * [new tag] ciflow/rocm/169470 -> ciflow/rocm/169470 2025-12-04T09:23:40.1336390Z * [new tag] ciflow/rocm/169471 -> ciflow/rocm/169471 2025-12-04T09:23:40.1336727Z * [new tag] ciflow/rocm/169472 -> ciflow/rocm/169472 2025-12-04T09:23:40.1336867Z * [new tag] ciflow/rocm/169514 -> ciflow/rocm/169514 2025-12-04T09:23:40.1337110Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-12-04T09:23:40.1337247Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-12-04T09:23:40.1340416Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-12-04T09:23:40.1340576Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-12-04T09:23:40.1340726Z * [new tag] ciflow/slow/167207 -> ciflow/slow/167207 2025-12-04T09:23:40.1340853Z * [new tag] ciflow/slow/168050 -> ciflow/slow/168050 2025-12-04T09:23:40.1340972Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-12-04T09:23:40.1341119Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-12-04T09:23:40.1342794Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-12-04T09:23:40.1342962Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-12-04T09:23:40.1343394Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-12-04T09:23:40.1343728Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-12-04T09:23:40.1346513Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-12-04T09:23:40.1346957Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-12-04T09:23:40.1347475Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-12-04T09:23:40.1347601Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-12-04T09:23:40.1347967Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-12-04T09:23:40.1348594Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-12-04T09:23:40.1348751Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-12-04T09:23:40.1348884Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-12-04T09:23:40.1349745Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-12-04T09:23:40.1350011Z * [new tag] ciflow/torchbench/168175 -> ciflow/torchbench/168175 2025-12-04T09:23:40.1353862Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-12-04T09:23:40.1354195Z * [new tag] ciflow/trunk/157149 -> ciflow/trunk/157149 2025-12-04T09:23:40.1354355Z * [new tag] ciflow/trunk/157994 -> ciflow/trunk/157994 2025-12-04T09:23:40.1354506Z * [new tag] ciflow/trunk/159718 -> ciflow/trunk/159718 2025-12-04T09:23:40.1354757Z * [new tag] ciflow/trunk/160685 -> ciflow/trunk/160685 2025-12-04T09:23:40.1354868Z * [new tag] ciflow/trunk/160729 -> ciflow/trunk/160729 2025-12-04T09:23:40.1354985Z * [new tag] ciflow/trunk/162275 -> ciflow/trunk/162275 2025-12-04T09:23:40.1355221Z * [new tag] ciflow/trunk/162795 -> ciflow/trunk/162795 2025-12-04T09:23:40.1355897Z * [new tag] ciflow/trunk/163245 -> ciflow/trunk/163245 2025-12-04T09:23:40.1356059Z * [new tag] ciflow/trunk/163942 -> ciflow/trunk/163942 2025-12-04T09:23:40.1356317Z * [new tag] ciflow/trunk/165274 -> ciflow/trunk/165274 2025-12-04T09:23:40.1356441Z * [new tag] ciflow/trunk/165483 -> ciflow/trunk/165483 2025-12-04T09:23:40.1356777Z * [new tag] ciflow/trunk/165728 -> ciflow/trunk/165728 2025-12-04T09:23:40.1357185Z * [new tag] ciflow/trunk/165922 -> ciflow/trunk/165922 2025-12-04T09:23:40.1357509Z * [new tag] ciflow/trunk/166075 -> ciflow/trunk/166075 2025-12-04T09:23:40.1358085Z * [new tag] ciflow/trunk/166165 -> ciflow/trunk/166165 2025-12-04T09:23:40.1358530Z * [new tag] ciflow/trunk/166829 -> ciflow/trunk/166829 2025-12-04T09:23:40.1359085Z * [new tag] ciflow/trunk/166843 -> ciflow/trunk/166843 2025-12-04T09:23:40.1359602Z * [new tag] ciflow/trunk/166876 -> ciflow/trunk/166876 2025-12-04T09:23:40.1360132Z * [new tag] ciflow/trunk/167207 -> ciflow/trunk/167207 2025-12-04T09:23:40.1360581Z * [new tag] ciflow/trunk/167536 -> ciflow/trunk/167536 2025-12-04T09:23:40.1361106Z * [new tag] ciflow/trunk/167552 -> ciflow/trunk/167552 2025-12-04T09:23:40.1361571Z * [new tag] ciflow/trunk/167555 -> ciflow/trunk/167555 2025-12-04T09:23:40.1362062Z * [new tag] ciflow/trunk/167599 -> ciflow/trunk/167599 2025-12-04T09:23:40.1365123Z * [new tag] ciflow/trunk/167659 -> ciflow/trunk/167659 2025-12-04T09:23:40.1365444Z * [new tag] ciflow/trunk/167672 -> ciflow/trunk/167672 2025-12-04T09:23:40.1365589Z * [new tag] ciflow/trunk/167742 -> ciflow/trunk/167742 2025-12-04T09:23:40.1365899Z * [new tag] ciflow/trunk/167781 -> ciflow/trunk/167781 2025-12-04T09:23:40.1366164Z * [new tag] ciflow/trunk/167837 -> ciflow/trunk/167837 2025-12-04T09:23:40.1366302Z * [new tag] ciflow/trunk/167887 -> ciflow/trunk/167887 2025-12-04T09:23:40.1366503Z * [new tag] ciflow/trunk/167978 -> ciflow/trunk/167978 2025-12-04T09:23:40.1367290Z * [new tag] ciflow/trunk/168050 -> ciflow/trunk/168050 2025-12-04T09:23:40.1367447Z * [new tag] ciflow/trunk/168051 -> ciflow/trunk/168051 2025-12-04T09:23:40.1367920Z * [new tag] ciflow/trunk/168096 -> ciflow/trunk/168096 2025-12-04T09:23:40.1368056Z * [new tag] ciflow/trunk/168127 -> ciflow/trunk/168127 2025-12-04T09:23:40.1368461Z * [new tag] ciflow/trunk/168157 -> ciflow/trunk/168157 2025-12-04T09:23:40.1368855Z * [new tag] ciflow/trunk/168175 -> ciflow/trunk/168175 2025-12-04T09:23:40.1369377Z * [new tag] ciflow/trunk/168209 -> ciflow/trunk/168209 2025-12-04T09:23:40.1369926Z * [new tag] ciflow/trunk/168213 -> ciflow/trunk/168213 2025-12-04T09:23:40.1370376Z * [new tag] ciflow/trunk/168226 -> ciflow/trunk/168226 2025-12-04T09:23:40.1371093Z * [new tag] ciflow/trunk/168262 -> ciflow/trunk/168262 2025-12-04T09:23:40.1371328Z * [new tag] ciflow/trunk/168275 -> ciflow/trunk/168275 2025-12-04T09:23:40.1373957Z * [new tag] ciflow/trunk/168328 -> ciflow/trunk/168328 2025-12-04T09:23:40.1374104Z * [new tag] ciflow/trunk/168368 -> ciflow/trunk/168368 2025-12-04T09:23:40.1374249Z * [new tag] ciflow/trunk/168917 -> ciflow/trunk/168917 2025-12-04T09:23:40.1374360Z * [new tag] ciflow/trunk/168933 -> ciflow/trunk/168933 2025-12-04T09:23:40.1374488Z * [new tag] ciflow/trunk/168941 -> ciflow/trunk/168941 2025-12-04T09:23:40.1374943Z * [new tag] ciflow/trunk/168955 -> ciflow/trunk/168955 2025-12-04T09:23:40.1375062Z * [new tag] ciflow/trunk/168980 -> ciflow/trunk/168980 2025-12-04T09:23:40.1375246Z * [new tag] ciflow/trunk/169004 -> ciflow/trunk/169004 2025-12-04T09:23:40.1376730Z * [new tag] ciflow/trunk/169006 -> ciflow/trunk/169006 2025-12-04T09:23:40.1376883Z * [new tag] ciflow/trunk/169023 -> ciflow/trunk/169023 2025-12-04T09:23:40.1377026Z * [new tag] ciflow/trunk/169025 -> ciflow/trunk/169025 2025-12-04T09:23:40.1377142Z * [new tag] ciflow/trunk/169048 -> ciflow/trunk/169048 2025-12-04T09:23:40.1380103Z * [new tag] ciflow/trunk/169066 -> ciflow/trunk/169066 2025-12-04T09:23:40.1380409Z * [new tag] ciflow/trunk/169091 -> ciflow/trunk/169091 2025-12-04T09:23:40.1380572Z * [new tag] ciflow/trunk/169102 -> ciflow/trunk/169102 2025-12-04T09:23:40.1380769Z * [new tag] ciflow/trunk/169103 -> ciflow/trunk/169103 2025-12-04T09:23:40.1385063Z * [new tag] ciflow/trunk/169125 -> ciflow/trunk/169125 2025-12-04T09:23:40.1385392Z * [new tag] ciflow/trunk/169139 -> ciflow/trunk/169139 2025-12-04T09:23:40.1385529Z * [new tag] ciflow/trunk/169148 -> ciflow/trunk/169148 2025-12-04T09:23:40.1385739Z * [new tag] ciflow/trunk/169151 -> ciflow/trunk/169151 2025-12-04T09:23:40.1385869Z * [new tag] ciflow/trunk/169156 -> ciflow/trunk/169156 2025-12-04T09:23:40.1385981Z * [new tag] ciflow/trunk/169176 -> ciflow/trunk/169176 2025-12-04T09:23:40.1386226Z * [new tag] ciflow/trunk/169204 -> ciflow/trunk/169204 2025-12-04T09:23:40.1386943Z * [new tag] ciflow/trunk/169207 -> ciflow/trunk/169207 2025-12-04T09:23:40.1387104Z * [new tag] ciflow/trunk/169211 -> ciflow/trunk/169211 2025-12-04T09:23:40.1387229Z * [new tag] ciflow/trunk/169231 -> ciflow/trunk/169231 2025-12-04T09:23:40.1387489Z * [new tag] ciflow/trunk/169260 -> ciflow/trunk/169260 2025-12-04T09:23:40.1387613Z * [new tag] ciflow/trunk/169271 -> ciflow/trunk/169271 2025-12-04T09:23:40.1387735Z * [new tag] ciflow/trunk/169280 -> ciflow/trunk/169280 2025-12-04T09:23:40.1387846Z * [new tag] ciflow/trunk/169281 -> ciflow/trunk/169281 2025-12-04T09:23:40.1387964Z * [new tag] ciflow/trunk/169286 -> ciflow/trunk/169286 2025-12-04T09:23:40.1388073Z * [new tag] ciflow/trunk/169293 -> ciflow/trunk/169293 2025-12-04T09:23:40.1388193Z * [new tag] ciflow/trunk/169296 -> ciflow/trunk/169296 2025-12-04T09:23:40.1388315Z * [new tag] ciflow/trunk/169304 -> ciflow/trunk/169304 2025-12-04T09:23:40.1388425Z * [new tag] ciflow/trunk/169305 -> ciflow/trunk/169305 2025-12-04T09:23:40.1388536Z * [new tag] ciflow/trunk/169312 -> ciflow/trunk/169312 2025-12-04T09:23:40.1394270Z * [new tag] ciflow/trunk/169328 -> ciflow/trunk/169328 2025-12-04T09:23:40.1394590Z * [new tag] ciflow/trunk/169343 -> ciflow/trunk/169343 2025-12-04T09:23:40.1394759Z * [new tag] ciflow/trunk/169355 -> ciflow/trunk/169355 2025-12-04T09:23:40.1394877Z * [new tag] ciflow/trunk/169370 -> ciflow/trunk/169370 2025-12-04T09:23:40.1395004Z * [new tag] ciflow/trunk/169379 -> ciflow/trunk/169379 2025-12-04T09:23:40.1395126Z * [new tag] ciflow/trunk/169380 -> ciflow/trunk/169380 2025-12-04T09:23:40.1395390Z * [new tag] ciflow/trunk/169385 -> ciflow/trunk/169385 2025-12-04T09:23:40.1395537Z * [new tag] ciflow/trunk/169387 -> ciflow/trunk/169387 2025-12-04T09:23:40.1395735Z * [new tag] ciflow/trunk/169410 -> ciflow/trunk/169410 2025-12-04T09:23:40.1395884Z * [new tag] ciflow/trunk/169412 -> ciflow/trunk/169412 2025-12-04T09:23:40.1396091Z * [new tag] ciflow/trunk/169418 -> ciflow/trunk/169418 2025-12-04T09:23:40.1396227Z * [new tag] ciflow/trunk/169423 -> ciflow/trunk/169423 2025-12-04T09:23:40.1396829Z * [new tag] ciflow/trunk/169427 -> ciflow/trunk/169427 2025-12-04T09:23:40.1397004Z * [new tag] ciflow/trunk/169430 -> ciflow/trunk/169430 2025-12-04T09:23:40.1397132Z * [new tag] ciflow/trunk/169437 -> ciflow/trunk/169437 2025-12-04T09:23:40.1397280Z * [new tag] ciflow/trunk/169442 -> ciflow/trunk/169442 2025-12-04T09:23:40.1397399Z * [new tag] ciflow/trunk/169452 -> ciflow/trunk/169452 2025-12-04T09:23:40.1397526Z * [new tag] ciflow/trunk/169454 -> ciflow/trunk/169454 2025-12-04T09:23:40.1397674Z * [new tag] ciflow/trunk/169459 -> ciflow/trunk/169459 2025-12-04T09:23:40.1398392Z * [new tag] ciflow/trunk/169474 -> ciflow/trunk/169474 2025-12-04T09:23:40.1398629Z * [new tag] ciflow/trunk/169475 -> ciflow/trunk/169475 2025-12-04T09:23:40.1398759Z * [new tag] ciflow/trunk/169476 -> ciflow/trunk/169476 2025-12-04T09:23:40.1399164Z * [new tag] ciflow/trunk/169487 -> ciflow/trunk/169487 2025-12-04T09:23:40.1399633Z * [new tag] ciflow/trunk/169497 -> ciflow/trunk/169497 2025-12-04T09:23:40.1400296Z * [new tag] ciflow/trunk/169503 -> ciflow/trunk/169503 2025-12-04T09:23:40.1400765Z * [new tag] ciflow/trunk/169505 -> ciflow/trunk/169505 2025-12-04T09:23:40.1401230Z * [new tag] ciflow/trunk/169507 -> ciflow/trunk/169507 2025-12-04T09:23:40.1401631Z * [new tag] ciflow/trunk/169514 -> ciflow/trunk/169514 2025-12-04T09:23:40.1405542Z * [new tag] ciflow/trunk/169517 -> ciflow/trunk/169517 2025-12-04T09:23:40.1405717Z * [new tag] ciflow/trunk/169519 -> ciflow/trunk/169519 2025-12-04T09:23:40.1405864Z * [new tag] ciflow/trunk/169528 -> ciflow/trunk/169528 2025-12-04T09:23:40.1405990Z * [new tag] ciflow/trunk/169541 -> ciflow/trunk/169541 2025-12-04T09:23:40.1406106Z * [new tag] ciflow/trunk/169555 -> ciflow/trunk/169555 2025-12-04T09:23:40.1406225Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-12-04T09:23:40.1406364Z * [new tag] ciflow/vllm/165270 -> ciflow/vllm/165270 2025-12-04T09:23:40.1406484Z * [new tag] ciflow/vllm/165274 -> ciflow/vllm/165274 2025-12-04T09:23:40.1406612Z * [new tag] ciflow/vllm/166494 -> ciflow/vllm/166494 2025-12-04T09:23:40.1406732Z * [new tag] ciflow/vllm/169219 -> ciflow/vllm/169219 2025-12-04T09:23:40.1407072Z * [new tag] ciflow/vllm/169220 -> ciflow/vllm/169220 2025-12-04T09:23:40.1407955Z * [new tag] ciflow/xpu/157994 -> ciflow/xpu/157994 2025-12-04T09:23:40.1408090Z * [new tag] ciflow/xpu/159718 -> ciflow/xpu/159718 2025-12-04T09:23:40.1408605Z * [new tag] ciflow/xpu/161940 -> ciflow/xpu/161940 2025-12-04T09:23:40.1411113Z * [new tag] ciflow/xpu/163251 -> ciflow/xpu/163251 2025-12-04T09:23:40.1411287Z * [new tag] ciflow/xpu/166829 -> ciflow/xpu/166829 2025-12-04T09:23:40.1411408Z * [new tag] ciflow/xpu/166843 -> ciflow/xpu/166843 2025-12-04T09:23:40.1411538Z * [new tag] ciflow/xpu/167972 -> ciflow/xpu/167972 2025-12-04T09:23:40.1411651Z * [new tag] ciflow/xpu/167981 -> ciflow/xpu/167981 2025-12-04T09:23:40.1411976Z * [new tag] ciflow/xpu/168213 -> ciflow/xpu/168213 2025-12-04T09:23:40.1412253Z * [new tag] ciflow/xpu/168262 -> ciflow/xpu/168262 2025-12-04T09:23:40.1412820Z * [new tag] ciflow/xpu/168328 -> ciflow/xpu/168328 2025-12-04T09:23:40.1413845Z * [new tag] ciflow/xpu/168950 -> ciflow/xpu/168950 2025-12-04T09:23:40.1414128Z * [new tag] ciflow/xpu/169039 -> ciflow/xpu/169039 2025-12-04T09:23:40.1415044Z * [new tag] ciflow/xpu/169200 -> ciflow/xpu/169200 2025-12-04T09:23:40.1415190Z * [new tag] ciflow/xpu/169203 -> ciflow/xpu/169203 2025-12-04T09:23:40.1415621Z * [new tag] ciflow/xpu/169230 -> ciflow/xpu/169230 2025-12-04T09:23:40.1416107Z * [new tag] ciflow/xpu/169231 -> ciflow/xpu/169231 2025-12-04T09:23:40.1419063Z * [new tag] ciflow/xpu/169241 -> ciflow/xpu/169241 2025-12-04T09:23:40.1419239Z * [new tag] ciflow/xpu/169280 -> ciflow/xpu/169280 2025-12-04T09:23:40.1419365Z * [new tag] ciflow/xpu/169296 -> ciflow/xpu/169296 2025-12-04T09:23:40.1419493Z * [new tag] ciflow/xpu/169353 -> ciflow/xpu/169353 2025-12-04T09:23:40.1419625Z * [new tag] ciflow/xpu/169410 -> ciflow/xpu/169410 2025-12-04T09:23:40.1419749Z * [new tag] ciflow/xpu/169442 -> ciflow/xpu/169442 2025-12-04T09:23:40.1419916Z * [new tag] ciflow/xpu/169555 -> ciflow/xpu/169555 2025-12-04T09:23:40.1420540Z * [new tag] cslpull75 -> cslpull75 2025-12-04T09:23:40.1424243Z * [new tag] cslpull76 -> cslpull76 2025-12-04T09:23:40.1424541Z * [new tag] cslpull77 -> cslpull77 2025-12-04T09:23:40.1424808Z * [new tag] cslpull78 -> cslpull78 2025-12-04T09:23:40.1425370Z * [new tag] cslpull79 -> cslpull79 2025-12-04T09:23:40.1425518Z * [new tag] cslpull80 -> cslpull80 2025-12-04T09:23:40.1425629Z * [new tag] cslpull81 -> cslpull81 2025-12-04T09:23:40.1425738Z * [new tag] cslpull82 -> cslpull82 2025-12-04T09:23:40.1425853Z * [new tag] cslpull83 -> cslpull83 2025-12-04T09:23:40.1425956Z * [new tag] cslpull84 -> cslpull84 2025-12-04T09:23:40.1426460Z * [new tag] cslpull85 -> cslpull85 2025-12-04T09:23:40.1431287Z * [new tag] cslpull86 -> cslpull86 2025-12-04T09:23:40.1433499Z * [new tag] cslpull87 -> cslpull87 2025-12-04T09:23:40.1433759Z * [new tag] cslpull88 -> cslpull88 2025-12-04T09:23:40.1434039Z * [new tag] cslpull89 -> cslpull89 2025-12-04T09:23:40.1434143Z * [new tag] cslpull90 -> cslpull90 2025-12-04T09:23:40.1434245Z * [new tag] cslpull91 -> cslpull91 2025-12-04T09:23:40.1434352Z * [new tag] cslpull92 -> cslpull92 2025-12-04T09:23:40.1434462Z * [new tag] flight_5 -> flight_5 2025-12-04T09:23:40.1434579Z * [new tag] flight_5.1 -> flight_5.1 2025-12-04T09:23:40.1434689Z * [new tag] flight_5.2 -> flight_5.2 2025-12-04T09:23:40.1434790Z * [new tag] flight_5.3 -> flight_5.3 2025-12-04T09:23:40.1434909Z * [new tag] forpull1 -> forpull1 2025-12-04T09:23:40.1435043Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-12-04T09:23:40.1435425Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-12-04T09:23:40.1436403Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-12-04T09:23:40.1437106Z * [new tag] nightly-binary -> nightly-binary 2025-12-04T09:23:40.1437558Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-12-04T09:23:40.1438352Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-12-04T09:23:40.1439568Z * [new tag] trunk/02d8bd6974cf84b721680d773dbdb1b6f40ce272 -> trunk/02d8bd6974cf84b721680d773dbdb1b6f40ce272 2025-12-04T09:23:40.1439847Z * [new tag] trunk/066997fb38ade71e00d78e9d572e380b5f02bd3e -> trunk/066997fb38ade71e00d78e9d572e380b5f02bd3e 2025-12-04T09:23:40.1441035Z * [new tag] trunk/076e7b19fa1d481ad778d06d2b49ba57d3ce8c88 -> trunk/076e7b19fa1d481ad778d06d2b49ba57d3ce8c88 2025-12-04T09:23:40.1441302Z * [new tag] trunk/07dcc0b83db3211653a38565a24e15acdba75654 -> trunk/07dcc0b83db3211653a38565a24e15acdba75654 2025-12-04T09:23:40.1444263Z * [new tag] trunk/082e96b68dfcd16cab7cfafc4d3d055767dab3eb -> trunk/082e96b68dfcd16cab7cfafc4d3d055767dab3eb 2025-12-04T09:23:40.1444557Z * [new tag] trunk/088048f2fea28ff7d450f65c72419ca45780d30b -> trunk/088048f2fea28ff7d450f65c72419ca45780d30b 2025-12-04T09:23:40.1444813Z * [new tag] trunk/09076941a95c76f4d9ad189d064dfd8baa39e672 -> trunk/09076941a95c76f4d9ad189d064dfd8baa39e672 2025-12-04T09:23:40.1445254Z * [new tag] trunk/0b80a4c62b94402844bf221791c096b0035c6d75 -> trunk/0b80a4c62b94402844bf221791c096b0035c6d75 2025-12-04T09:23:40.1445506Z * [new tag] trunk/0bbbdf1750567a980634ad907a325357ba8ba8f2 -> trunk/0bbbdf1750567a980634ad907a325357ba8ba8f2 2025-12-04T09:23:40.1445817Z * [new tag] trunk/0c281dd78773b2bc17c58ead0e4cd4ac46e775c5 -> trunk/0c281dd78773b2bc17c58ead0e4cd4ac46e775c5 2025-12-04T09:23:40.1446080Z * [new tag] trunk/135f3753c418a6879b1954904184937b67e61688 -> trunk/135f3753c418a6879b1954904184937b67e61688 2025-12-04T09:23:40.1446403Z * [new tag] trunk/15da21026cb13cd20257dc9e96830db108743c10 -> trunk/15da21026cb13cd20257dc9e96830db108743c10 2025-12-04T09:23:40.1452557Z * [new tag] trunk/166efdad2ac827f30fb02504c6017520257f88ec -> trunk/166efdad2ac827f30fb02504c6017520257f88ec 2025-12-04T09:23:40.1452841Z * [new tag] trunk/174272c15fae553d8488140af931f7d8050a313f -> trunk/174272c15fae553d8488140af931f7d8050a313f 2025-12-04T09:23:40.1453123Z * [new tag] trunk/18f3ca08f13b8de61307f5e8cd7d4cccb67e9d11 -> trunk/18f3ca08f13b8de61307f5e8cd7d4cccb67e9d11 2025-12-04T09:23:40.1453386Z * [new tag] trunk/1902eddfe655a15ebcf2c72bd81ade110fdeef63 -> trunk/1902eddfe655a15ebcf2c72bd81ade110fdeef63 2025-12-04T09:23:40.1453641Z * [new tag] trunk/195f92e98d3d66738577f11f22c4b5c8a1c76dd5 -> trunk/195f92e98d3d66738577f11f22c4b5c8a1c76dd5 2025-12-04T09:23:40.1453907Z * [new tag] trunk/1aa13e17de39e3c768ea7aebaad166ce72a06676 -> trunk/1aa13e17de39e3c768ea7aebaad166ce72a06676 2025-12-04T09:23:40.1454214Z * [new tag] trunk/1afe2832f58e24e54a5bfda5a5afa9b96fdea40e -> trunk/1afe2832f58e24e54a5bfda5a5afa9b96fdea40e 2025-12-04T09:23:40.1454490Z * [new tag] trunk/1c87554d74140eaee964ca8b1832cede67f5f520 -> trunk/1c87554d74140eaee964ca8b1832cede67f5f520 2025-12-04T09:23:40.1454748Z * [new tag] trunk/1ccb743b7b5be955f49736c162c4f5004b8a0dd8 -> trunk/1ccb743b7b5be955f49736c162c4f5004b8a0dd8 2025-12-04T09:23:40.1455025Z * [new tag] trunk/1cee47d6ce0a02227185b566593f002dd639ca0c -> trunk/1cee47d6ce0a02227185b566593f002dd639ca0c 2025-12-04T09:23:40.1455299Z * [new tag] trunk/1d21b4df2babe322e5d085ceb6de884eb260a62d -> trunk/1d21b4df2babe322e5d085ceb6de884eb260a62d 2025-12-04T09:23:40.1455543Z * [new tag] trunk/1e34fb2550e4aa650314f7a6d9f6daf4da7478a8 -> trunk/1e34fb2550e4aa650314f7a6d9f6daf4da7478a8 2025-12-04T09:23:40.1455985Z * [new tag] trunk/1e526fb5b1d93bfc70691c5c3955fdffc1b7b7de -> trunk/1e526fb5b1d93bfc70691c5c3955fdffc1b7b7de 2025-12-04T09:23:40.1456366Z * [new tag] trunk/1ee32a8b1f554a312d79bad01ded24f38cd95543 -> trunk/1ee32a8b1f554a312d79bad01ded24f38cd95543 2025-12-04T09:23:40.1456737Z * [new tag] trunk/201e2c4117eb9744594dad6a5c18213d7b4705d7 -> trunk/201e2c4117eb9744594dad6a5c18213d7b4705d7 2025-12-04T09:23:40.1457512Z * [new tag] trunk/2353a0f60eb4b4cb6675907a7fa9fbedc1c02e7f -> trunk/2353a0f60eb4b4cb6675907a7fa9fbedc1c02e7f 2025-12-04T09:23:40.1457958Z * [new tag] trunk/285779b1621cf9f073a062b0889a642d200308d9 -> trunk/285779b1621cf9f073a062b0889a642d200308d9 2025-12-04T09:23:40.1458360Z * [new tag] trunk/2887faaec6295d081580d09fce161201826c6d87 -> trunk/2887faaec6295d081580d09fce161201826c6d87 2025-12-04T09:23:40.1458661Z * [new tag] trunk/296e67c92635443c67b11c0ae1bd045f03ebb7bc -> trunk/296e67c92635443c67b11c0ae1bd045f03ebb7bc 2025-12-04T09:23:40.1459275Z * [new tag] trunk/29856679769b3dede478767e2fe6cfb51197cb25 -> trunk/29856679769b3dede478767e2fe6cfb51197cb25 2025-12-04T09:23:40.1459868Z * [new tag] trunk/29e5455a4740c326ab187c7aa7b5ef98034ea563 -> trunk/29e5455a4740c326ab187c7aa7b5ef98034ea563 2025-12-04T09:23:40.1460445Z * [new tag] trunk/2ac3ef882afb23136adc188975f0a8802fc68adf -> trunk/2ac3ef882afb23136adc188975f0a8802fc68adf 2025-12-04T09:23:40.1461413Z * [new tag] trunk/2bec68e73b64715354af076ad309335f943e36cd -> trunk/2bec68e73b64715354af076ad309335f943e36cd 2025-12-04T09:23:40.1461866Z * [new tag] trunk/2c87367e6f88662cd5cedbd1537748b7948c38e1 -> trunk/2c87367e6f88662cd5cedbd1537748b7948c38e1 2025-12-04T09:23:40.1462471Z * [new tag] trunk/2d1f78fe3ec13820f136a2e0336da12a25f41708 -> trunk/2d1f78fe3ec13820f136a2e0336da12a25f41708 2025-12-04T09:23:40.1462765Z * [new tag] trunk/2df6058f116a65722a0e03073402feb242572d35 -> trunk/2df6058f116a65722a0e03073402feb242572d35 2025-12-04T09:23:40.1463619Z * [new tag] trunk/2e0c2e170fe658c440775c8e5c44228aafcc47ec -> trunk/2e0c2e170fe658c440775c8e5c44228aafcc47ec 2025-12-04T09:23:40.1464426Z * [new tag] trunk/2f9b7dad7b5419b063bd0f2e204de192720ebb94 -> trunk/2f9b7dad7b5419b063bd0f2e204de192720ebb94 2025-12-04T09:23:40.1464870Z * [new tag] trunk/305168768a95d69c444df5cd334bb774edfe06f1 -> trunk/305168768a95d69c444df5cd334bb774edfe06f1 2025-12-04T09:23:40.1465213Z * [new tag] trunk/31fc12773026e8e00f054dd79ad9b2491e693b48 -> trunk/31fc12773026e8e00f054dd79ad9b2491e693b48 2025-12-04T09:23:40.1465698Z * [new tag] trunk/320de0c6b0a3e7c6d2693ea5c28d5d0156ba7991 -> trunk/320de0c6b0a3e7c6d2693ea5c28d5d0156ba7991 2025-12-04T09:23:40.1470834Z * [new tag] trunk/3418bd29475dff06695045fcdf93e7d0dac67da8 -> trunk/3418bd29475dff06695045fcdf93e7d0dac67da8 2025-12-04T09:23:40.1473017Z * [new tag] trunk/34a98608afa0cb5b48f0d6d30432fdd0a2614ddf -> trunk/34a98608afa0cb5b48f0d6d30432fdd0a2614ddf 2025-12-04T09:23:40.1473414Z * [new tag] trunk/35b7a9a26c5923d98aebaa41a031dae21788a9ee -> trunk/35b7a9a26c5923d98aebaa41a031dae21788a9ee 2025-12-04T09:23:40.1476754Z * [new tag] trunk/39d07dbf03a911bdd45d1af78d8638dc92074938 -> trunk/39d07dbf03a911bdd45d1af78d8638dc92074938 2025-12-04T09:23:40.1477206Z * [new tag] trunk/3cd98b4205ada151042cc7ff097a82d4a4b18725 -> trunk/3cd98b4205ada151042cc7ff097a82d4a4b18725 2025-12-04T09:23:40.1477516Z * [new tag] trunk/3d35fd20a78ff4d016fa80f4e5fad37191d7bcae -> trunk/3d35fd20a78ff4d016fa80f4e5fad37191d7bcae 2025-12-04T09:23:40.1477908Z * [new tag] trunk/409a5fee945c46a3edaf5df162812f201bfd7b2f -> trunk/409a5fee945c46a3edaf5df162812f201bfd7b2f 2025-12-04T09:23:40.1478165Z * [new tag] trunk/42e9005cda22da3f1c559c3649218cebd671027c -> trunk/42e9005cda22da3f1c559c3649218cebd671027c 2025-12-04T09:23:40.1478423Z * [new tag] trunk/43b94713bbf340d3c124fde02d0f73add4021247 -> trunk/43b94713bbf340d3c124fde02d0f73add4021247 2025-12-04T09:23:40.1478664Z * [new tag] trunk/44ac69388a4a5eb463dbd2a13f00d1e3b924566c -> trunk/44ac69388a4a5eb463dbd2a13f00d1e3b924566c 2025-12-04T09:23:40.1478921Z * [new tag] trunk/45d14e2497292be06ad36eaa1aaaf7c630a2586a -> trunk/45d14e2497292be06ad36eaa1aaaf7c630a2586a 2025-12-04T09:23:40.1479179Z * [new tag] trunk/45d310ad84854dff730c0b12e577d7998d978686 -> trunk/45d310ad84854dff730c0b12e577d7998d978686 2025-12-04T09:23:40.1479437Z * [new tag] trunk/47b28ddf7bd74b50fa93b307a7d3b183a6d77f54 -> trunk/47b28ddf7bd74b50fa93b307a7d3b183a6d77f54 2025-12-04T09:23:40.1479686Z * [new tag] trunk/481e5ab336275bd3acd5fa8a611b05b4469012af -> trunk/481e5ab336275bd3acd5fa8a611b05b4469012af 2025-12-04T09:23:40.1479931Z * [new tag] trunk/491731647f6b8a9345dcfb3bc9416aea254a7d96 -> trunk/491731647f6b8a9345dcfb3bc9416aea254a7d96 2025-12-04T09:23:40.1480179Z * [new tag] trunk/49a04d26088acc17d948ddd66920f3e16371e873 -> trunk/49a04d26088acc17d948ddd66920f3e16371e873 2025-12-04T09:23:40.1480651Z * [new tag] trunk/4bebc827c47d2f1f0fa1a417a5201a97aef3d985 -> trunk/4bebc827c47d2f1f0fa1a417a5201a97aef3d985 2025-12-04T09:23:40.1481062Z * [new tag] trunk/4c246677784c6a14bc2dbb9ff8773ef0a3a3222f -> trunk/4c246677784c6a14bc2dbb9ff8773ef0a3a3222f 2025-12-04T09:23:40.1481307Z * [new tag] trunk/4cfb47ff548b6d996641058cf04a70e311a4c3aa -> trunk/4cfb47ff548b6d996641058cf04a70e311a4c3aa 2025-12-04T09:23:40.1481702Z * [new tag] trunk/4e0061c1aa52f606dda8cfab0bd7591e588faf2c -> trunk/4e0061c1aa52f606dda8cfab0bd7591e588faf2c 2025-12-04T09:23:40.1481955Z * [new tag] trunk/4fefb8e7e942386ffac764a41b232241f82bea3a -> trunk/4fefb8e7e942386ffac764a41b232241f82bea3a 2025-12-04T09:23:40.1482195Z * [new tag] trunk/503b2640023521f5a35cd9a52fc8033d73a95d0d -> trunk/503b2640023521f5a35cd9a52fc8033d73a95d0d 2025-12-04T09:23:40.1482464Z * [new tag] trunk/518c2b1b3dab9a2ef2849e04b3bc2f20c1c41db9 -> trunk/518c2b1b3dab9a2ef2849e04b3bc2f20c1c41db9 2025-12-04T09:23:40.1482720Z * [new tag] trunk/5191b2fa68ba19960912bfd7fd721c79d76bb1f3 -> trunk/5191b2fa68ba19960912bfd7fd721c79d76bb1f3 2025-12-04T09:23:40.1482974Z * [new tag] trunk/52ac0f0dc4acacd219f1317fbc28ec631c01e07a -> trunk/52ac0f0dc4acacd219f1317fbc28ec631c01e07a 2025-12-04T09:23:40.1483209Z * [new tag] trunk/539ba711b029de9f191070f4f0d12f18f5b7f292 -> trunk/539ba711b029de9f191070f4f0d12f18f5b7f292 2025-12-04T09:23:40.1483460Z * [new tag] trunk/556375b55deebebbc56cb7aef81f4d52f031ba28 -> trunk/556375b55deebebbc56cb7aef81f4d52f031ba28 2025-12-04T09:23:40.1483855Z * [new tag] trunk/55c4ab554845481d0a69a3811937575fe8bb1a66 -> trunk/55c4ab554845481d0a69a3811937575fe8bb1a66 2025-12-04T09:23:40.1484178Z * [new tag] trunk/5634469fda9e5d98869c82c7d03bb08914245f96 -> trunk/5634469fda9e5d98869c82c7d03bb08914245f96 2025-12-04T09:23:40.1484522Z * [new tag] trunk/5778f6ff894686a975a9a23645178ae4c87ad5dc -> trunk/5778f6ff894686a975a9a23645178ae4c87ad5dc 2025-12-04T09:23:40.1484962Z * [new tag] trunk/587d63a3e07de5dc91065f9ef70bcacda9989068 -> trunk/587d63a3e07de5dc91065f9ef70bcacda9989068 2025-12-04T09:23:40.1485964Z * [new tag] trunk/597930f6b568852356ca9795dac76f9e4653adbd -> trunk/597930f6b568852356ca9795dac76f9e4653adbd 2025-12-04T09:23:40.1486269Z * [new tag] trunk/597df3a4e2a67b9fdbe1a89b2f4d74f822274db6 -> trunk/597df3a4e2a67b9fdbe1a89b2f4d74f822274db6 2025-12-04T09:23:40.1489713Z * [new tag] trunk/59abd50e931f4efb21b053f7a2911f5d8a49d883 -> trunk/59abd50e931f4efb21b053f7a2911f5d8a49d883 2025-12-04T09:23:40.1491754Z * [new tag] trunk/5a607febc04c3a2b5824c75f3f60307867439a2c -> trunk/5a607febc04c3a2b5824c75f3f60307867439a2c 2025-12-04T09:23:40.1492189Z * [new tag] trunk/5bf1cdf4755c54ef462b44cb8041b0a57311556b -> trunk/5bf1cdf4755c54ef462b44cb8041b0a57311556b 2025-12-04T09:23:40.1492568Z * [new tag] trunk/5f0030ba63d334d7e8c93a09e41403b89e4c573c -> trunk/5f0030ba63d334d7e8c93a09e41403b89e4c573c 2025-12-04T09:23:40.1492950Z * [new tag] trunk/5f21d27e71268464d362a96c9ac09ea475f7f202 -> trunk/5f21d27e71268464d362a96c9ac09ea475f7f202 2025-12-04T09:23:40.1493286Z * [new tag] trunk/5fafc13038c9988d9ac21fa793fbd5890604b447 -> trunk/5fafc13038c9988d9ac21fa793fbd5890604b447 2025-12-04T09:23:40.1494058Z * [new tag] trunk/61be54a31dc09b59d99b62176fb935aee0b924ef -> trunk/61be54a31dc09b59d99b62176fb935aee0b924ef 2025-12-04T09:23:40.1494353Z * [new tag] trunk/62d3ccd71484ed6a760d909b41487101bbc65719 -> trunk/62d3ccd71484ed6a760d909b41487101bbc65719 2025-12-04T09:23:40.1494610Z * [new tag] trunk/641cdb68ae27668eb441d0e49c87a0602c120c2b -> trunk/641cdb68ae27668eb441d0e49c87a0602c120c2b 2025-12-04T09:23:40.1494845Z * [new tag] trunk/65c4620d6bb0c6029f69762c22b91dda2294da9a -> trunk/65c4620d6bb0c6029f69762c22b91dda2294da9a 2025-12-04T09:23:40.1495097Z * [new tag] trunk/66004b993744b4106bf8afaba71f3c228a804206 -> trunk/66004b993744b4106bf8afaba71f3c228a804206 2025-12-04T09:23:40.1495461Z * [new tag] trunk/6658a04c7ca67acb64512341342e7b3ee13ee386 -> trunk/6658a04c7ca67acb64512341342e7b3ee13ee386 2025-12-04T09:23:40.1495703Z * [new tag] trunk/6864e309092a71f8ab0ca6a4dc7f8a4073fd31c4 -> trunk/6864e309092a71f8ab0ca6a4dc7f8a4073fd31c4 2025-12-04T09:23:40.1495996Z * [new tag] trunk/6c261c6cb07892c90ca19ed51c9705b1659a3f7d -> trunk/6c261c6cb07892c90ca19ed51c9705b1659a3f7d 2025-12-04T09:23:40.1496232Z * [new tag] trunk/6c8b6a043f1628188b6396b3a2a6e000ca68362b -> trunk/6c8b6a043f1628188b6396b3a2a6e000ca68362b 2025-12-04T09:23:40.1496781Z * [new tag] trunk/6ceb4a32f92ae67ce5d7d97931d17401ebf5ffa5 -> trunk/6ceb4a32f92ae67ce5d7d97931d17401ebf5ffa5 2025-12-04T09:23:40.1497225Z * [new tag] trunk/6e404e9b7d6f5fb0de86aa73888c3038248c17f8 -> trunk/6e404e9b7d6f5fb0de86aa73888c3038248c17f8 2025-12-04T09:23:40.1497917Z * [new tag] trunk/6ec30b490aee1db6bcdc7340abddef25784f08ec -> trunk/6ec30b490aee1db6bcdc7340abddef25784f08ec 2025-12-04T09:23:40.1498461Z * [new tag] trunk/6f2783a6c08e1db34275ff25176ffe9aebc30a71 -> trunk/6f2783a6c08e1db34275ff25176ffe9aebc30a71 2025-12-04T09:23:40.1503144Z * [new tag] trunk/6f53fefeb90ad3281119b5cfc4aa9ffd8a066e3d -> trunk/6f53fefeb90ad3281119b5cfc4aa9ffd8a066e3d 2025-12-04T09:23:40.1503442Z * [new tag] trunk/6f7dcf51e46d0c880db1a2f5c70de57adb576f4a -> trunk/6f7dcf51e46d0c880db1a2f5c70de57adb576f4a 2025-12-04T09:23:40.1503699Z * [new tag] trunk/6ff831180d2fa436c7f1c1af3adac641fce9d60e -> trunk/6ff831180d2fa436c7f1c1af3adac641fce9d60e 2025-12-04T09:23:40.1503938Z * [new tag] trunk/70076464a63ab218a7ceefb0e76ccd7131deb8f8 -> trunk/70076464a63ab218a7ceefb0e76ccd7131deb8f8 2025-12-04T09:23:40.1504171Z * [new tag] trunk/70d797a5fc109b20a517646fcaa819477cd0d485 -> trunk/70d797a5fc109b20a517646fcaa819477cd0d485 2025-12-04T09:23:40.1504420Z * [new tag] trunk/7348cb355ff0a6f79cd4871215aea72185748734 -> trunk/7348cb355ff0a6f79cd4871215aea72185748734 2025-12-04T09:23:40.1504656Z * [new tag] trunk/74fe26a1ebe32931783569f2e762e3c2c974901f -> trunk/74fe26a1ebe32931783569f2e762e3c2c974901f 2025-12-04T09:23:40.1504923Z * [new tag] trunk/76aeb8c7e0f795b3fddca134cbea9a69da3ee696 -> trunk/76aeb8c7e0f795b3fddca134cbea9a69da3ee696 2025-12-04T09:23:40.1505169Z * [new tag] trunk/7716da9fb23f27a65b41f9f016a2afadf281c18f -> trunk/7716da9fb23f27a65b41f9f016a2afadf281c18f 2025-12-04T09:23:40.1505398Z * [new tag] trunk/7741edd4ed665f3988052e260863efb508d61a03 -> trunk/7741edd4ed665f3988052e260863efb508d61a03 2025-12-04T09:23:40.1505645Z * [new tag] trunk/78adb3b3df41b45d2368b67226d2f864b78939a6 -> trunk/78adb3b3df41b45d2368b67226d2f864b78939a6 2025-12-04T09:23:40.1506605Z * [new tag] trunk/79d7b178225e5ed24d4e1db74e5abbff848f5fb7 -> trunk/79d7b178225e5ed24d4e1db74e5abbff848f5fb7 2025-12-04T09:23:40.1506861Z * [new tag] trunk/7a1e316115fc6996b3f2336822ba5d5f6179f0c3 -> trunk/7a1e316115fc6996b3f2336822ba5d5f6179f0c3 2025-12-04T09:23:40.1507097Z * [new tag] trunk/7a41b66367c38d0af3e8a90f7be48d6b281e7bca -> trunk/7a41b66367c38d0af3e8a90f7be48d6b281e7bca 2025-12-04T09:23:40.1507472Z * [new tag] trunk/7b7af390ea8541c611d1ce2018a6934188fc197b -> trunk/7b7af390ea8541c611d1ce2018a6934188fc197b 2025-12-04T09:23:40.1514390Z * [new tag] trunk/7ba4680f3755a560af81aa0f688791e367aa3609 -> trunk/7ba4680f3755a560af81aa0f688791e367aa3609 2025-12-04T09:23:40.1514867Z * [new tag] trunk/7bc2a66ded06a0b2549aa51d807edc5dc3e73d1b -> trunk/7bc2a66ded06a0b2549aa51d807edc5dc3e73d1b 2025-12-04T09:23:40.1515265Z * [new tag] trunk/7c648509a7470ace9fb2bae960dd4790f7e943e9 -> trunk/7c648509a7470ace9fb2bae960dd4790f7e943e9 2025-12-04T09:23:40.1516268Z * [new tag] trunk/7cbc2d034cecd21ab5c9707d0a9c525c17143fb8 -> trunk/7cbc2d034cecd21ab5c9707d0a9c525c17143fb8 2025-12-04T09:23:40.1516567Z * [new tag] trunk/7d1bbaf4ba301ea3fba6f3c7bc02d58f6417aaed -> trunk/7d1bbaf4ba301ea3fba6f3c7bc02d58f6417aaed 2025-12-04T09:23:40.1517082Z * [new tag] trunk/7d2a33e4ebf60b217a3cd77feae19231eb996fc8 -> trunk/7d2a33e4ebf60b217a3cd77feae19231eb996fc8 2025-12-04T09:23:40.1517337Z * [new tag] trunk/7eb625920054b1126a7d2d99818aaa188c6ba95e -> trunk/7eb625920054b1126a7d2d99818aaa188c6ba95e 2025-12-04T09:23:40.1517608Z * [new tag] trunk/7f55ba19c456a3d6cc443dd9edb6bb7cca677ead -> trunk/7f55ba19c456a3d6cc443dd9edb6bb7cca677ead 2025-12-04T09:23:40.1517852Z * [new tag] trunk/81af382128efa094d8702e18f2c133760904c718 -> trunk/81af382128efa094d8702e18f2c133760904c718 2025-12-04T09:23:40.1518102Z * [new tag] trunk/84149583d483e9c973c9a0feda70e4f3964947b0 -> trunk/84149583d483e9c973c9a0feda70e4f3964947b0 2025-12-04T09:23:40.1518372Z * [new tag] trunk/85a315917efe82c24306be805c584ec044951c75 -> trunk/85a315917efe82c24306be805c584ec044951c75 2025-12-04T09:23:40.1518616Z * [new tag] trunk/87329491c82a5f8c1cc4ec11d8f55a5de2551ece -> trunk/87329491c82a5f8c1cc4ec11d8f55a5de2551ece 2025-12-04T09:23:40.1518867Z * [new tag] trunk/892640e25aeefa8007c5af837214b4502b6b62a6 -> trunk/892640e25aeefa8007c5af837214b4502b6b62a6 2025-12-04T09:23:40.1519115Z * [new tag] trunk/89e3bbcb5b5321dc8b9520b4d5a8ee60cea1d0b4 -> trunk/89e3bbcb5b5321dc8b9520b4d5a8ee60cea1d0b4 2025-12-04T09:23:40.1519366Z * [new tag] trunk/8c73bbbb02159223c0c97d268a0a74cb78158a1c -> trunk/8c73bbbb02159223c0c97d268a0a74cb78158a1c 2025-12-04T09:23:40.1519620Z * [new tag] trunk/8d56e98c8db988a22cb2dfaeefb30bc7d2a3cc43 -> trunk/8d56e98c8db988a22cb2dfaeefb30bc7d2a3cc43 2025-12-04T09:23:40.1519868Z * [new tag] trunk/8d9dd9603e5ee26c01007f0cd4f018e584840922 -> trunk/8d9dd9603e5ee26c01007f0cd4f018e584840922 2025-12-04T09:23:40.1520132Z * [new tag] trunk/8ef0c0b02b062d75e7c9be2594914a3e784d23ca -> trunk/8ef0c0b02b062d75e7c9be2594914a3e784d23ca 2025-12-04T09:23:40.1520585Z * [new tag] trunk/90b27e7e8352cde97d32ddad24740ef819633f38 -> trunk/90b27e7e8352cde97d32ddad24740ef819633f38 2025-12-04T09:23:40.1520840Z * [new tag] trunk/90f0139e64b2951815d524b6a373bed20c4fbf90 -> trunk/90f0139e64b2951815d524b6a373bed20c4fbf90 2025-12-04T09:23:40.1521086Z * [new tag] trunk/93d0d6838c56af59b0dba794e6aa08f0c1c7799c -> trunk/93d0d6838c56af59b0dba794e6aa08f0c1c7799c 2025-12-04T09:23:40.1521338Z * [new tag] trunk/94ca8d5f1e81fea3ae488650a0fb6795049a9f87 -> trunk/94ca8d5f1e81fea3ae488650a0fb6795049a9f87 2025-12-04T09:23:40.1521593Z * [new tag] trunk/9844fbeadd5cebdf1281d6fbf79164139c352693 -> trunk/9844fbeadd5cebdf1281d6fbf79164139c352693 2025-12-04T09:23:40.1521843Z * [new tag] trunk/99024dec888ec1e50b546822a32b6fb2f35e5eaa -> trunk/99024dec888ec1e50b546822a32b6fb2f35e5eaa 2025-12-04T09:23:40.1531158Z * [new tag] trunk/9a296e640fc88aa44d275b48cd9cc30c573b169d -> trunk/9a296e640fc88aa44d275b48cd9cc30c573b169d 2025-12-04T09:23:40.1532974Z * [new tag] trunk/9b3e34d8589b29f7b4e7fab6f78711b7ca6e4639 -> trunk/9b3e34d8589b29f7b4e7fab6f78711b7ca6e4639 2025-12-04T09:23:40.1533252Z * [new tag] trunk/9cd055e547e9b67a5f9827f8999c38d7eda1bcb8 -> trunk/9cd055e547e9b67a5f9827f8999c38d7eda1bcb8 2025-12-04T09:23:40.1533527Z * [new tag] trunk/9f0df5686cb4ada94f94620acba2e3c3f363b11d -> trunk/9f0df5686cb4ada94f94620acba2e3c3f363b11d 2025-12-04T09:23:40.1533767Z * [new tag] trunk/9f7fceb887d0cfa0326a59b887821c63ff11340a -> trunk/9f7fceb887d0cfa0326a59b887821c63ff11340a 2025-12-04T09:23:40.1534005Z * [new tag] trunk/9f8ef8855d3078d70f7b782540ff2aaf158d6742 -> trunk/9f8ef8855d3078d70f7b782540ff2aaf158d6742 2025-12-04T09:23:40.1534409Z * [new tag] trunk/9fb52efc797b47a1f425a03aa5e47b866d8b1098 -> trunk/9fb52efc797b47a1f425a03aa5e47b866d8b1098 2025-12-04T09:23:40.1534846Z * [new tag] trunk/9ff4a2ebc5762d46c73e46b1b523d7ff349fedfa -> trunk/9ff4a2ebc5762d46c73e46b1b523d7ff349fedfa 2025-12-04T09:23:40.1535296Z * [new tag] trunk/a0f3937b94422354538ebbd47202d5b0e8a3fd0d -> trunk/a0f3937b94422354538ebbd47202d5b0e8a3fd0d 2025-12-04T09:23:40.1535683Z * [new tag] trunk/a15066c28b3145e6edbfc88359d0411d14cfc70c -> trunk/a15066c28b3145e6edbfc88359d0411d14cfc70c 2025-12-04T09:23:40.1536368Z * [new tag] trunk/a20f775e82564d2a9979221ed7f3b8d7cf54ce90 -> trunk/a20f775e82564d2a9979221ed7f3b8d7cf54ce90 2025-12-04T09:23:40.1536682Z * [new tag] trunk/a2973fb00ec002dd4b6bbf07385f066efb259b8c -> trunk/a2973fb00ec002dd4b6bbf07385f066efb259b8c 2025-12-04T09:23:40.1536965Z * [new tag] trunk/a7dc6dab9ad911259d4801c502907e531594db45 -> trunk/a7dc6dab9ad911259d4801c502907e531594db45 2025-12-04T09:23:40.1537225Z * [new tag] trunk/a951a9cee65c01660bbc6e6fded90ecb10fa6109 -> trunk/a951a9cee65c01660bbc6e6fded90ecb10fa6109 2025-12-04T09:23:40.1537479Z * [new tag] trunk/abfa1a6d65c7c159e35c72c25979b9da4971689e -> trunk/abfa1a6d65c7c159e35c72c25979b9da4971689e 2025-12-04T09:23:40.1537725Z * [new tag] trunk/ae3a2395bf66151078e2d201716f7d63ce1c6f3e -> trunk/ae3a2395bf66151078e2d201716f7d63ce1c6f3e 2025-12-04T09:23:40.1537994Z * [new tag] trunk/afdff7f0325080dedac44d080cb5a3b0e65e6c5e -> trunk/afdff7f0325080dedac44d080cb5a3b0e65e6c5e 2025-12-04T09:23:40.1538245Z * [new tag] trunk/b1aed4e7a72c03a38f44543aaea0dae2e9b76d48 -> trunk/b1aed4e7a72c03a38f44543aaea0dae2e9b76d48 2025-12-04T09:23:40.1538502Z * [new tag] trunk/b1decff555cd50e2123c8c6e25cc0d447c411f62 -> trunk/b1decff555cd50e2123c8c6e25cc0d447c411f62 2025-12-04T09:23:40.1538754Z * [new tag] trunk/b2b6b034c9fd08672c40e63ef243556ad4c49bd2 -> trunk/b2b6b034c9fd08672c40e63ef243556ad4c49bd2 2025-12-04T09:23:40.1538982Z * [new tag] trunk/b39813b4a04931682b0491adba2138d01d716d99 -> trunk/b39813b4a04931682b0491adba2138d01d716d99 2025-12-04T09:23:40.1539232Z * [new tag] trunk/b3a7edb2311367974cc7cd764cfb11a5d6758b24 -> trunk/b3a7edb2311367974cc7cd764cfb11a5d6758b24 2025-12-04T09:23:40.1539649Z * [new tag] trunk/b4cc1329c86acaef6d42c1fac7169b8d870ab0d7 -> trunk/b4cc1329c86acaef6d42c1fac7169b8d870ab0d7 2025-12-04T09:23:40.1539988Z * [new tag] trunk/b555c39217f765759954a4f9f9bd1e9b87bed11a -> trunk/b555c39217f765759954a4f9f9bd1e9b87bed11a 2025-12-04T09:23:40.1540307Z * [new tag] trunk/b6b6c80379388b7f9932c3e6a0f9907bf430e417 -> trunk/b6b6c80379388b7f9932c3e6a0f9907bf430e417 2025-12-04T09:23:40.1540744Z * [new tag] trunk/b6b6d912df0b6f4082f8e50b18bd1de1dd7325f4 -> trunk/b6b6d912df0b6f4082f8e50b18bd1de1dd7325f4 2025-12-04T09:23:40.1540995Z * [new tag] trunk/b7d60685f8cbc939b68a20871e90db67e729329b -> trunk/b7d60685f8cbc939b68a20871e90db67e729329b 2025-12-04T09:23:40.1541245Z * [new tag] trunk/b7f6b9a4fc6259f7af068f31868b3119bb1bac3e -> trunk/b7f6b9a4fc6259f7af068f31868b3119bb1bac3e 2025-12-04T09:23:40.1541489Z * [new tag] trunk/b8c4ba3593761e7b2a3ebd86f040fb07b47c02cf -> trunk/b8c4ba3593761e7b2a3ebd86f040fb07b47c02cf 2025-12-04T09:23:40.1541727Z * [new tag] trunk/b9c8f3a4884befb965ff42620ce44a71b04887f5 -> trunk/b9c8f3a4884befb965ff42620ce44a71b04887f5 2025-12-04T09:23:40.1541966Z * [new tag] trunk/ba1412546f3082c0958c077acc2025e4dbc33f1f -> trunk/ba1412546f3082c0958c077acc2025e4dbc33f1f 2025-12-04T09:23:40.1542212Z * [new tag] trunk/bac403c0b38c63bdbcc0c31f1c2b0bc0260f610f -> trunk/bac403c0b38c63bdbcc0c31f1c2b0bc0260f610f 2025-12-04T09:23:40.1542730Z * [new tag] trunk/bb3034198b459401fabeab254e1b99f0115046e2 -> trunk/bb3034198b459401fabeab254e1b99f0115046e2 2025-12-04T09:23:40.1543068Z * [new tag] trunk/bc39b2b3bc7a6e19a42e62bd576974035086fe55 -> trunk/bc39b2b3bc7a6e19a42e62bd576974035086fe55 2025-12-04T09:23:40.1543471Z * [new tag] trunk/bc43d5b297f207a11d83d77ddf0152bdaabe15a8 -> trunk/bc43d5b297f207a11d83d77ddf0152bdaabe15a8 2025-12-04T09:23:40.1543855Z * [new tag] trunk/bc6a4863c7246a6493d16d4ea6eee71ec07c6a09 -> trunk/bc6a4863c7246a6493d16d4ea6eee71ec07c6a09 2025-12-04T09:23:40.1544196Z * [new tag] trunk/bea4912944defdbcb8b061800caab6cbbbd01df5 -> trunk/bea4912944defdbcb8b061800caab6cbbbd01df5 2025-12-04T09:23:40.1544536Z * [new tag] trunk/c04e2c656f48d82d1521b867bbbf03967b9b7564 -> trunk/c04e2c656f48d82d1521b867bbbf03967b9b7564 2025-12-04T09:23:40.1545238Z * [new tag] trunk/c0660bcee27e7d7731634e274576a7081882bede -> trunk/c0660bcee27e7d7731634e274576a7081882bede 2025-12-04T09:23:40.1550768Z * [new tag] trunk/c178ed43d3d99cbefe84fbfb21d6f282b20d62ac -> trunk/c178ed43d3d99cbefe84fbfb21d6f282b20d62ac 2025-12-04T09:23:40.1551218Z * [new tag] trunk/c55b1e8f61d041ee436d697449eb028931d574fb -> trunk/c55b1e8f61d041ee436d697449eb028931d574fb 2025-12-04T09:23:40.1551651Z * [new tag] trunk/c6ae7579fe12fe75f1a8f7043a494c90567273f1 -> trunk/c6ae7579fe12fe75f1a8f7043a494c90567273f1 2025-12-04T09:23:40.1552060Z * [new tag] trunk/c8210e7d94bad5ae21ac389fa4ba8a463c76c4d0 -> trunk/c8210e7d94bad5ae21ac389fa4ba8a463c76c4d0 2025-12-04T09:23:40.1552312Z * [new tag] trunk/cc0853af42122f8185321f542616f4474e717f09 -> trunk/cc0853af42122f8185321f542616f4474e717f09 2025-12-04T09:23:40.1552579Z * [new tag] trunk/cddec6562eabfa390d014fa3741a5659cf9c94c9 -> trunk/cddec6562eabfa390d014fa3741a5659cf9c94c9 2025-12-04T09:23:40.1552856Z * [new tag] trunk/ce5e7e3bf1f4b69a4f4f93d288ba75b906df492a -> trunk/ce5e7e3bf1f4b69a4f4f93d288ba75b906df492a 2025-12-04T09:23:40.1553111Z * [new tag] trunk/d038b0130ec7c20ebcac219301292fd8e98a1ace -> trunk/d038b0130ec7c20ebcac219301292fd8e98a1ace 2025-12-04T09:23:40.1553342Z * [new tag] trunk/d16447dacaf2420ea175f0c275c75da951f57d39 -> trunk/d16447dacaf2420ea175f0c275c75da951f57d39 2025-12-04T09:23:40.1553565Z * [new tag] trunk/d19f1e8cab6810bb2e99141f9976665954c67a50 -> trunk/d19f1e8cab6810bb2e99141f9976665954c67a50 2025-12-04T09:23:40.1553802Z * [new tag] trunk/d1c9f03b2a5af4104721712f8cdffe9b4f340c01 -> trunk/d1c9f03b2a5af4104721712f8cdffe9b4f340c01 2025-12-04T09:23:40.1554031Z * [new tag] trunk/d40f4950f2b7f7aa380a22fe0f6166e71680fbcf -> trunk/d40f4950f2b7f7aa380a22fe0f6166e71680fbcf 2025-12-04T09:23:40.1554277Z * [new tag] trunk/d5038950bacfe36bbf24a47a455fe76901deb8e8 -> trunk/d5038950bacfe36bbf24a47a455fe76901deb8e8 2025-12-04T09:23:40.1554528Z * [new tag] trunk/d54ff42903c2ae0533931ff11d23b35f875bdb3d -> trunk/d54ff42903c2ae0533931ff11d23b35f875bdb3d 2025-12-04T09:23:40.1554765Z * [new tag] trunk/d76697633a2d2b9cced1ae21161849b33bfe7e47 -> trunk/d76697633a2d2b9cced1ae21161849b33bfe7e47 2025-12-04T09:23:40.1555020Z * [new tag] trunk/d78f52b199c547106d4cd9d2856dd0805c118bf1 -> trunk/d78f52b199c547106d4cd9d2856dd0805c118bf1 2025-12-04T09:23:40.1557700Z * [new tag] trunk/d8fd5c6eed28e5004150691d048a3f6785e19a8e -> trunk/d8fd5c6eed28e5004150691d048a3f6785e19a8e 2025-12-04T09:23:40.1557988Z * [new tag] trunk/d900f5e86745dec76713f4b0ef07005ef36b2f5a -> trunk/d900f5e86745dec76713f4b0ef07005ef36b2f5a 2025-12-04T09:23:40.1558255Z * [new tag] trunk/d973dc6b87d763859fe1c5bd1287e3b6b1c49d1b -> trunk/d973dc6b87d763859fe1c5bd1287e3b6b1c49d1b 2025-12-04T09:23:40.1558744Z * [new tag] trunk/d998c03304cb6ede76e1ed535b4ddeb6c2bf40ec -> trunk/d998c03304cb6ede76e1ed535b4ddeb6c2bf40ec 2025-12-04T09:23:40.1559008Z * [new tag] trunk/d9cb8a70833101dbbe16b99520cfbdd70d0a87bf -> trunk/d9cb8a70833101dbbe16b99520cfbdd70d0a87bf 2025-12-04T09:23:40.1559390Z * [new tag] trunk/d9d5e91b43f70eb8637af55db6856d49be391ffd -> trunk/d9d5e91b43f70eb8637af55db6856d49be391ffd 2025-12-04T09:23:40.1559634Z * [new tag] trunk/dd18a75336a4fbd7497955cc5665904724fce889 -> trunk/dd18a75336a4fbd7497955cc5665904724fce889 2025-12-04T09:23:40.1559896Z * [new tag] trunk/ded9bcd61a059bf723e6e84689552962b480ea77 -> trunk/ded9bcd61a059bf723e6e84689552962b480ea77 2025-12-04T09:23:40.1560158Z * [new tag] trunk/dfbd3714d15c37a7b83b322a6b60f997fc00f50c -> trunk/dfbd3714d15c37a7b83b322a6b60f997fc00f50c 2025-12-04T09:23:40.1560637Z * [new tag] trunk/e115f9f4e4b039f8e9a642aaa2bd8254a920541b -> trunk/e115f9f4e4b039f8e9a642aaa2bd8254a920541b 2025-12-04T09:23:40.1560906Z * [new tag] trunk/e3f24fd73ad74c6e7176687986436956c7c18235 -> trunk/e3f24fd73ad74c6e7176687986436956c7c18235 2025-12-04T09:23:40.1561160Z * [new tag] trunk/e7d24d3ff93d1503ba63860b7057438ad93f918e -> trunk/e7d24d3ff93d1503ba63860b7057438ad93f918e 2025-12-04T09:23:40.1561416Z * [new tag] trunk/ea7035f462a0d2830865ee86c832bd101e1427fc -> trunk/ea7035f462a0d2830865ee86c832bd101e1427fc 2025-12-04T09:23:40.1561866Z * [new tag] trunk/eabb7ad2128580ef674446027b95bcf4e21e8df3 -> trunk/eabb7ad2128580ef674446027b95bcf4e21e8df3 2025-12-04T09:23:40.1562163Z * [new tag] trunk/eb5c63652a33da42e7018c23df5f20a3eb4c6ccf -> trunk/eb5c63652a33da42e7018c23df5f20a3eb4c6ccf 2025-12-04T09:23:40.1565406Z * [new tag] trunk/ec2c71f5c85021b8938cdafadce24c15a36fd93e -> trunk/ec2c71f5c85021b8938cdafadce24c15a36fd93e 2025-12-04T09:23:40.1565867Z * [new tag] trunk/ecbcc3f6bf327856b435b259ac63cc2f328c4b4e -> trunk/ecbcc3f6bf327856b435b259ac63cc2f328c4b4e 2025-12-04T09:23:40.1566221Z * [new tag] trunk/ee87bbe876c42575e961b32a0827d76bc9782ca2 -> trunk/ee87bbe876c42575e961b32a0827d76bc9782ca2 2025-12-04T09:23:40.1566581Z * [new tag] trunk/ef019d1d431c4c5a95b594cb90d40a50cd00f5e4 -> trunk/ef019d1d431c4c5a95b594cb90d40a50cd00f5e4 2025-12-04T09:23:40.1566912Z * [new tag] trunk/ef8ecc13830a86c4b231f1aad9aba7851db61b53 -> trunk/ef8ecc13830a86c4b231f1aad9aba7851db61b53 2025-12-04T09:23:40.1567612Z * [new tag] trunk/f1076f5510920044912247b1abb8760cb820f598 -> trunk/f1076f5510920044912247b1abb8760cb820f598 2025-12-04T09:23:40.1567938Z * [new tag] trunk/f2d6a75a00a1d648ca9a0abc6a33e14c3dea6c40 -> trunk/f2d6a75a00a1d648ca9a0abc6a33e14c3dea6c40 2025-12-04T09:23:40.1568216Z * [new tag] trunk/f47dd0ddef1359e5b43e4b962412f67b30ecde56 -> trunk/f47dd0ddef1359e5b43e4b962412f67b30ecde56 2025-12-04T09:23:40.1568515Z * [new tag] trunk/f49d32dfa4730dcfb1b60eeeb369b5889da983c8 -> trunk/f49d32dfa4730dcfb1b60eeeb369b5889da983c8 2025-12-04T09:23:40.1569228Z * [new tag] trunk/f4dedf78fc30fd4b93975787ca6074ee89db9467 -> trunk/f4dedf78fc30fd4b93975787ca6074ee89db9467 2025-12-04T09:23:40.1569539Z * [new tag] trunk/f7c0d03819ebed05c4038f095d66d1b8c54aca17 -> trunk/f7c0d03819ebed05c4038f095d66d1b8c54aca17 2025-12-04T09:23:40.1572638Z * [new tag] trunk/f7e1bd80a063e17453c361837ba6ea2570920a73 -> trunk/f7e1bd80a063e17453c361837ba6ea2570920a73 2025-12-04T09:23:40.1572930Z * [new tag] trunk/f9bd6c53624c7c0ea3772de78498326e84c2f0e7 -> trunk/f9bd6c53624c7c0ea3772de78498326e84c2f0e7 2025-12-04T09:23:40.1573186Z * [new tag] trunk/fb5be221a46b51bfc9509013b0d85bc5a9d4f15b -> trunk/fb5be221a46b51bfc9509013b0d85bc5a9d4f15b 2025-12-04T09:23:40.1573443Z * [new tag] trunk/fdf863d5e1de3b2688c9511e96876e34581dbfd7 -> trunk/fdf863d5e1de3b2688c9511e96876e34581dbfd7 2025-12-04T09:23:40.1573847Z * [new tag] trunk/fe0e65adfc0e7ca6e5f57e6ea8b16bd5cc967307 -> trunk/fe0e65adfc0e7ca6e5f57e6ea8b16bd5cc967307 2025-12-04T09:23:40.1578904Z * [new tag] trunk/fec710bf89173f5355468a7ce1afe9157c3d9009 -> trunk/fec710bf89173f5355468a7ce1afe9157c3d9009 2025-12-04T09:23:40.1579365Z * [new tag] trunk/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 -> trunk/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:23:40.1579535Z * [new tag] v0.1.1 -> v0.1.1 2025-12-04T09:23:40.1579652Z * [new tag] v0.1.10 -> v0.1.10 2025-12-04T09:23:40.1579756Z * [new tag] v0.1.11 -> v0.1.11 2025-12-04T09:23:40.1579866Z * [new tag] v0.1.12 -> v0.1.12 2025-12-04T09:23:40.1579973Z * [new tag] v0.1.2 -> v0.1.2 2025-12-04T09:23:40.1580262Z * [new tag] v0.1.3 -> v0.1.3 2025-12-04T09:23:40.1580373Z * [new tag] v0.1.4 -> v0.1.4 2025-12-04T09:23:40.1580470Z * [new tag] v0.1.5 -> v0.1.5 2025-12-04T09:23:40.1580574Z * [new tag] v0.1.6 -> v0.1.6 2025-12-04T09:23:40.1580674Z * [new tag] v0.1.7 -> v0.1.7 2025-12-04T09:23:40.1580770Z * [new tag] v0.1.8 -> v0.1.8 2025-12-04T09:23:40.1581028Z * [new tag] v0.1.9 -> v0.1.9 2025-12-04T09:23:40.1585269Z * [new tag] v0.2.0 -> v0.2.0 2025-12-04T09:23:40.1585975Z * [new tag] v0.3.0 -> v0.3.0 2025-12-04T09:23:40.1586242Z * [new tag] v0.3.1 -> v0.3.1 2025-12-04T09:23:40.1586369Z * [new tag] v0.4.0 -> v0.4.0 2025-12-04T09:23:40.1586572Z * [new tag] v0.4.1 -> v0.4.1 2025-12-04T09:23:40.1586699Z * [new tag] v1.0.0 -> v1.0.0 2025-12-04T09:23:40.1586832Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-12-04T09:23:40.1590362Z * [new tag] v1.0.1 -> v1.0.1 2025-12-04T09:23:40.1590533Z * [new tag] v1.0rc0 -> v1.0rc0 2025-12-04T09:23:40.1590638Z * [new tag] v1.0rc1 -> v1.0rc1 2025-12-04T09:23:40.1590740Z * [new tag] v1.1.0 -> v1.1.0 2025-12-04T09:23:40.1590867Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-12-04T09:23:40.1590966Z * [new tag] v1.10.0 -> v1.10.0 2025-12-04T09:23:40.1591087Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-12-04T09:23:40.1591216Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-12-04T09:23:40.1591316Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-12-04T09:23:40.1591421Z * [new tag] v1.10.1 -> v1.10.1 2025-12-04T09:23:40.1591522Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-12-04T09:23:40.1591629Z * [new tag] v1.10.2 -> v1.10.2 2025-12-04T09:23:40.1591730Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-12-04T09:23:40.1596815Z * [new tag] v1.11.0 -> v1.11.0 2025-12-04T09:23:40.1596987Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-12-04T09:23:40.1597203Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-12-04T09:23:40.1597867Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-12-04T09:23:40.1598388Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-12-04T09:23:40.1598711Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-12-04T09:23:40.1598816Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-12-04T09:23:40.1598919Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-12-04T09:23:40.1599088Z * [new tag] v1.12.0 -> v1.12.0 2025-12-04T09:23:40.1599195Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-12-04T09:23:40.1599302Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-12-04T09:23:40.1599404Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-12-04T09:23:40.1599504Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-12-04T09:23:40.1599626Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-12-04T09:23:40.1599728Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-12-04T09:23:40.1599846Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-12-04T09:23:40.1599950Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-12-04T09:23:40.1600060Z * [new tag] v1.12.1 -> v1.12.1 2025-12-04T09:23:40.1600176Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-12-04T09:23:40.1600633Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-12-04T09:23:40.1608527Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-12-04T09:23:40.1608664Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-12-04T09:23:40.1608783Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-12-04T09:23:40.1608890Z * [new tag] v1.13.0 -> v1.13.0 2025-12-04T09:23:40.1609011Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-12-04T09:23:40.1609129Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-12-04T09:23:40.1609229Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-12-04T09:23:40.1609338Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-12-04T09:23:40.1609434Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-12-04T09:23:40.1609558Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-12-04T09:23:40.1609688Z * [new tag] v1.13.1 -> v1.13.1 2025-12-04T09:23:40.1610003Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-12-04T09:23:40.1610278Z * [new tag] v1.2.0 -> v1.2.0 2025-12-04T09:23:40.1610404Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-12-04T09:23:40.1610504Z * [new tag] v1.3.0 -> v1.3.0 2025-12-04T09:23:40.1610625Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-12-04T09:23:40.1610721Z * [new tag] v1.3.1 -> v1.3.1 2025-12-04T09:23:40.1610815Z * [new tag] v1.4.0 -> v1.4.0 2025-12-04T09:23:40.1610921Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-12-04T09:23:40.1611024Z * [new tag] v1.4.1 -> v1.4.1 2025-12-04T09:23:40.1611249Z * [new tag] v1.5.0 -> v1.5.0 2025-12-04T09:23:40.1617815Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-12-04T09:23:40.1618353Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-12-04T09:23:40.1618681Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-12-04T09:23:40.1618793Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-12-04T09:23:40.1619016Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-12-04T09:23:40.1619354Z * [new tag] v1.5.1 -> v1.5.1 2025-12-04T09:23:40.1619499Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-12-04T09:23:40.1619695Z * [new tag] v1.6.0 -> v1.6.0 2025-12-04T09:23:40.1619915Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-12-04T09:23:40.1620070Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-12-04T09:23:40.1620177Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-12-04T09:23:40.1620278Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-12-04T09:23:40.1620512Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-12-04T09:23:40.1620664Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-12-04T09:23:40.1620763Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-12-04T09:23:40.1620874Z * [new tag] v1.7.0 -> v1.7.0 2025-12-04T09:23:40.1620972Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-12-04T09:23:40.1624887Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-12-04T09:23:40.1625393Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-12-04T09:23:40.1625575Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-12-04T09:23:40.1625689Z * [new tag] v1.7.1 -> v1.7.1 2025-12-04T09:23:40.1625805Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-12-04T09:23:40.1625907Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-12-04T09:23:40.1626009Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-12-04T09:23:40.1628693Z * [new tag] v1.8.0 -> v1.8.0 2025-12-04T09:23:40.1629001Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-12-04T09:23:40.1629465Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-12-04T09:23:40.1629583Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-12-04T09:23:40.1629742Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-12-04T09:23:40.1629968Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-12-04T09:23:40.1630595Z * [new tag] v1.8.1 -> v1.8.1 2025-12-04T09:23:40.1630754Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-12-04T09:23:40.1631083Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-12-04T09:23:40.1631201Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-12-04T09:23:40.1634029Z * [new tag] v1.8.2 -> v1.8.2 2025-12-04T09:23:40.1634317Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-12-04T09:23:40.1634487Z * [new tag] v1.9.0 -> v1.9.0 2025-12-04T09:23:40.1634747Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-12-04T09:23:40.1635024Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-12-04T09:23:40.1635157Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-12-04T09:23:40.1635305Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-12-04T09:23:40.1635455Z * [new tag] v1.9.1 -> v1.9.1 2025-12-04T09:23:40.1637535Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-12-04T09:23:40.1637685Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-12-04T09:23:40.1637808Z * [new tag] v2.0.0 -> v2.0.0 2025-12-04T09:23:40.1637934Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-12-04T09:23:40.1638229Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-12-04T09:23:40.1638341Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-12-04T09:23:40.1638446Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-12-04T09:23:40.1638562Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-12-04T09:23:40.1638738Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-12-04T09:23:40.1638849Z * [new tag] v2.0.1 -> v2.0.1 2025-12-04T09:23:40.1639340Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-12-04T09:23:40.1639619Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-12-04T09:23:40.1640053Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-12-04T09:23:40.1641400Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-12-04T09:23:40.1648427Z * [new tag] v2.1.0 -> v2.1.0 2025-12-04T09:23:40.1648593Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-12-04T09:23:40.1648705Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-12-04T09:23:40.1648808Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-12-04T09:23:40.1648933Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-12-04T09:23:40.1649036Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-12-04T09:23:40.1649134Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-12-04T09:23:40.1649687Z * [new tag] v2.1.1 -> v2.1.1 2025-12-04T09:23:40.1649793Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-12-04T09:23:40.1649905Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-12-04T09:23:40.1650010Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-12-04T09:23:40.1650117Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-12-04T09:23:40.1650225Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-12-04T09:23:40.1650486Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-12-04T09:23:40.1650653Z * [new tag] v2.1.2 -> v2.1.2 2025-12-04T09:23:40.1656384Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-12-04T09:23:40.1656699Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-12-04T09:23:40.1656838Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-12-04T09:23:40.1656961Z * [new tag] v2.2.0 -> v2.2.0 2025-12-04T09:23:40.1657068Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-12-04T09:23:40.1657174Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-12-04T09:23:40.1657437Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-12-04T09:23:40.1657699Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-12-04T09:23:40.1658327Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-12-04T09:23:40.1658471Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-12-04T09:23:40.1658896Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-12-04T09:23:40.1659023Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-12-04T09:23:40.1659132Z * [new tag] v2.2.1 -> v2.2.1 2025-12-04T09:23:40.1659253Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-12-04T09:23:40.1659352Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-12-04T09:23:40.1659452Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-12-04T09:23:40.1659770Z * [new tag] v2.2.2 -> v2.2.2 2025-12-04T09:23:40.1664278Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-12-04T09:23:40.1664524Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-12-04T09:23:40.1664769Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-12-04T09:23:40.1665103Z * [new tag] v2.3.0 -> v2.3.0 2025-12-04T09:23:40.1665355Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-12-04T09:23:40.1665482Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-12-04T09:23:40.1665628Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-12-04T09:23:40.1665737Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-12-04T09:23:40.1665937Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-12-04T09:23:40.1666051Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-12-04T09:23:40.1666183Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-12-04T09:23:40.1667471Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-12-04T09:23:40.1667760Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-12-04T09:23:40.1667909Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-12-04T09:23:40.1668099Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-12-04T09:23:40.1668308Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-12-04T09:23:40.1668416Z * [new tag] v2.3.1 -> v2.3.1 2025-12-04T09:23:40.1668649Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-12-04T09:23:40.1668774Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-12-04T09:23:40.1673599Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-12-04T09:23:40.1673920Z * [new tag] v2.4.0 -> v2.4.0 2025-12-04T09:23:40.1674306Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-12-04T09:23:40.1674481Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-12-04T09:23:40.1674653Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-12-04T09:23:40.1674796Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-12-04T09:23:40.1675355Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-12-04T09:23:40.1675505Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-12-04T09:23:40.1675623Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-12-04T09:23:40.1675728Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-12-04T09:23:40.1675830Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-12-04T09:23:40.1675969Z * [new tag] v2.4.1 -> v2.4.1 2025-12-04T09:23:40.1676073Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-12-04T09:23:40.1676181Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-12-04T09:23:40.1676986Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-12-04T09:23:40.1677116Z * [new tag] v2.5.0 -> v2.5.0 2025-12-04T09:23:40.1677593Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-12-04T09:23:40.1684543Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-12-04T09:23:40.1684859Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-12-04T09:23:40.1684988Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-12-04T09:23:40.1685097Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-12-04T09:23:40.1685324Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-12-04T09:23:40.1685678Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-12-04T09:23:40.1685799Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-12-04T09:23:40.1685902Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-12-04T09:23:40.1690363Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-12-04T09:23:40.1690687Z * [new tag] v2.5.1 -> v2.5.1 2025-12-04T09:23:40.1690826Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-12-04T09:23:40.1690964Z * [new tag] v2.6.0 -> v2.6.0 2025-12-04T09:23:40.1691092Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-12-04T09:23:40.1691242Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-12-04T09:23:40.1691361Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-12-04T09:23:40.1691615Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-12-04T09:23:40.1692231Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-12-04T09:23:40.1692501Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-12-04T09:23:40.1692639Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-12-04T09:23:40.1692792Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-12-04T09:23:40.1692914Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-12-04T09:23:40.1693065Z * [new tag] v2.7.0 -> v2.7.0 2025-12-04T09:23:40.1693184Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-12-04T09:23:40.1693317Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-12-04T09:23:40.1698375Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-12-04T09:23:40.1698531Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-12-04T09:23:40.1698676Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-12-04T09:23:40.1698858Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-12-04T09:23:40.1699059Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-12-04T09:23:40.1699186Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-12-04T09:23:40.1699426Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-12-04T09:23:40.1699614Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-12-04T09:23:40.1699726Z * [new tag] v2.7.1 -> v2.7.1 2025-12-04T09:23:40.1699965Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-12-04T09:23:40.1700083Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-12-04T09:23:40.1700280Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-12-04T09:23:40.1700398Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-12-04T09:23:40.1700545Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-12-04T09:23:40.1700652Z * [new tag] v2.8.0 -> v2.8.0 2025-12-04T09:23:40.1703686Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-12-04T09:23:40.1708185Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-12-04T09:23:40.1710057Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-12-04T09:23:40.1710319Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-12-04T09:23:40.1714955Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-12-04T09:23:40.1715303Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-12-04T09:23:40.1715442Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-12-04T09:23:40.1715700Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-12-04T09:23:40.1715955Z * [new tag] v2.9.0 -> v2.9.0 2025-12-04T09:23:40.1716105Z * [new tag] v2.9.0-rc1 -> v2.9.0-rc1 2025-12-04T09:23:40.1716278Z * [new tag] v2.9.0-rc10 -> v2.9.0-rc10 2025-12-04T09:23:40.1716387Z * [new tag] v2.9.0-rc11 -> v2.9.0-rc11 2025-12-04T09:23:40.1716492Z * [new tag] v2.9.0-rc2 -> v2.9.0-rc2 2025-12-04T09:23:40.1716605Z * [new tag] v2.9.0-rc3 -> v2.9.0-rc3 2025-12-04T09:23:40.1716711Z * [new tag] v2.9.0-rc4 -> v2.9.0-rc4 2025-12-04T09:23:40.1716825Z * [new tag] v2.9.0-rc5 -> v2.9.0-rc5 2025-12-04T09:23:40.1716933Z * [new tag] v2.9.0-rc6 -> v2.9.0-rc6 2025-12-04T09:23:40.1717047Z * [new tag] v2.9.0-rc7 -> v2.9.0-rc7 2025-12-04T09:23:40.1717173Z * [new tag] v2.9.0-rc8 -> v2.9.0-rc8 2025-12-04T09:23:40.1717283Z * [new tag] v2.9.0-rc9 -> v2.9.0-rc9 2025-12-04T09:23:40.1717396Z * [new tag] v2.9.1 -> v2.9.1 2025-12-04T09:23:40.1717501Z * [new tag] v2.9.1-rc1 -> v2.9.1-rc1 2025-12-04T09:23:40.1717602Z * [new tag] v2.9.1-rc2 -> v2.9.1-rc2 2025-12-04T09:23:40.1717753Z * [new tag] viable/strict/1759343184 -> viable/strict/1759343184 2025-12-04T09:23:40.1717891Z * [new tag] viable/strict/1759346540 -> viable/strict/1759346540 2025-12-04T09:23:40.1718035Z * [new tag] viable/strict/1759348181 -> viable/strict/1759348181 2025-12-04T09:23:40.1718169Z * [new tag] viable/strict/1759350324 -> viable/strict/1759350324 2025-12-04T09:23:40.1718304Z * [new tag] viable/strict/1759351793 -> viable/strict/1759351793 2025-12-04T09:23:40.1718445Z * [new tag] viable/strict/1759353844 -> viable/strict/1759353844 2025-12-04T09:23:40.1718716Z * [new tag] viable/strict/1759355374 -> viable/strict/1759355374 2025-12-04T09:23:40.1718877Z * [new tag] viable/strict/1759357472 -> viable/strict/1759357472 2025-12-04T09:23:40.1719259Z * [new tag] viable/strict/1759361002 -> viable/strict/1759361002 2025-12-04T09:23:40.1720511Z * [new tag] viable/strict/1759362585 -> viable/strict/1759362585 2025-12-04T09:23:40.1720811Z * [new tag] viable/strict/1759365359 -> viable/strict/1759365359 2025-12-04T09:23:40.1727962Z * [new tag] viable/strict/1759370089 -> viable/strict/1759370089 2025-12-04T09:23:40.1728272Z * [new tag] viable/strict/1759377554 -> viable/strict/1759377554 2025-12-04T09:23:40.1728694Z * [new tag] viable/strict/1759379133 -> viable/strict/1759379133 2025-12-04T09:23:40.1728883Z * [new tag] viable/strict/1759389871 -> viable/strict/1759389871 2025-12-04T09:23:40.1729046Z * [new tag] viable/strict/1759393562 -> viable/strict/1759393562 2025-12-04T09:23:40.1729210Z * [new tag] viable/strict/1759395076 -> viable/strict/1759395076 2025-12-04T09:23:40.1729414Z * [new tag] viable/strict/1759398579 -> viable/strict/1759398579 2025-12-04T09:23:40.1729608Z * [new tag] viable/strict/1759404142 -> viable/strict/1759404142 2025-12-04T09:23:40.1729820Z * [new tag] viable/strict/1759405773 -> viable/strict/1759405773 2025-12-04T09:23:40.1730494Z * [new tag] viable/strict/1759408041 -> viable/strict/1759408041 2025-12-04T09:23:40.1730673Z * [new tag] viable/strict/1759411593 -> viable/strict/1759411593 2025-12-04T09:23:40.1730969Z * [new tag] viable/strict/1759427395 -> viable/strict/1759427395 2025-12-04T09:23:40.1731108Z * [new tag] viable/strict/1759434582 -> viable/strict/1759434582 2025-12-04T09:23:40.1731237Z * [new tag] viable/strict/1759436720 -> viable/strict/1759436720 2025-12-04T09:23:40.1731437Z * [new tag] viable/strict/1759440219 -> viable/strict/1759440219 2025-12-04T09:23:40.1731577Z * [new tag] viable/strict/1759441948 -> viable/strict/1759441948 2025-12-04T09:23:40.1733746Z * [new tag] viable/strict/1759443860 -> viable/strict/1759443860 2025-12-04T09:23:40.1733925Z * [new tag] viable/strict/1759445377 -> viable/strict/1759445377 2025-12-04T09:23:40.1734081Z * [new tag] viable/strict/1759447415 -> viable/strict/1759447415 2025-12-04T09:23:40.1734219Z * [new tag] viable/strict/1759451750 -> viable/strict/1759451750 2025-12-04T09:23:40.1734382Z * [new tag] viable/strict/1759453910 -> viable/strict/1759453910 2025-12-04T09:23:40.1734525Z * [new tag] viable/strict/1759456483 -> viable/strict/1759456483 2025-12-04T09:23:40.1734660Z * [new tag] viable/strict/1759459279 -> viable/strict/1759459279 2025-12-04T09:23:40.1734821Z * [new tag] viable/strict/1759460742 -> viable/strict/1759460742 2025-12-04T09:23:40.1740381Z * [new tag] viable/strict/1759462025 -> viable/strict/1759462025 2025-12-04T09:23:40.1740558Z * [new tag] viable/strict/1759469086 -> viable/strict/1759469086 2025-12-04T09:23:40.1740696Z * [new tag] viable/strict/1759470581 -> viable/strict/1759470581 2025-12-04T09:23:40.1740833Z * [new tag] viable/strict/1759472786 -> viable/strict/1759472786 2025-12-04T09:23:40.1740972Z * [new tag] viable/strict/1759476294 -> viable/strict/1759476294 2025-12-04T09:23:40.1741120Z * [new tag] viable/strict/1759479963 -> viable/strict/1759479963 2025-12-04T09:23:40.1741266Z * [new tag] viable/strict/1759492177 -> viable/strict/1759492177 2025-12-04T09:23:40.1741397Z * [new tag] viable/strict/1759519278 -> viable/strict/1759519278 2025-12-04T09:23:40.1741556Z * [new tag] viable/strict/1759524580 -> viable/strict/1759524580 2025-12-04T09:23:40.1741926Z * [new tag] viable/strict/1759528193 -> viable/strict/1759528193 2025-12-04T09:23:40.1742075Z * [new tag] viable/strict/1759533797 -> viable/strict/1759533797 2025-12-04T09:23:40.1742234Z * [new tag] viable/strict/1759542780 -> viable/strict/1759542780 2025-12-04T09:23:40.1742378Z * [new tag] viable/strict/1759549779 -> viable/strict/1759549779 2025-12-04T09:23:40.1742525Z * [new tag] viable/strict/1759555455 -> viable/strict/1759555455 2025-12-04T09:23:40.1742659Z * [new tag] viable/strict/1759559176 -> viable/strict/1759559176 2025-12-04T09:23:40.1742804Z * [new tag] viable/strict/1759560629 -> viable/strict/1759560629 2025-12-04T09:23:40.1748891Z * [new tag] viable/strict/1759569848 -> viable/strict/1759569848 2025-12-04T09:23:40.1749081Z * [new tag] viable/strict/1759571382 -> viable/strict/1759571382 2025-12-04T09:23:40.1749240Z * [new tag] viable/strict/1759573474 -> viable/strict/1759573474 2025-12-04T09:23:40.1749385Z * [new tag] viable/strict/1759618187 -> viable/strict/1759618187 2025-12-04T09:23:40.1749526Z * [new tag] viable/strict/1759626742 -> viable/strict/1759626742 2025-12-04T09:23:40.1749679Z * [new tag] viable/strict/1759632427 -> viable/strict/1759632427 2025-12-04T09:23:40.1749825Z * [new tag] viable/strict/1759634971 -> viable/strict/1759634971 2025-12-04T09:23:40.1750246Z * [new tag] viable/strict/1759661382 -> viable/strict/1759661382 2025-12-04T09:23:40.1750390Z * [new tag] viable/strict/1759663294 -> viable/strict/1759663294 2025-12-04T09:23:40.1751849Z * [new tag] viable/strict/1759708178 -> viable/strict/1759708178 2025-12-04T09:23:40.1756773Z * [new tag] viable/strict/1759715695 -> viable/strict/1759715695 2025-12-04T09:23:40.1757132Z * [new tag] viable/strict/1759728293 -> viable/strict/1759728293 2025-12-04T09:23:40.1757296Z * [new tag] viable/strict/1759735513 -> viable/strict/1759735513 2025-12-04T09:23:40.1757544Z * [new tag] viable/strict/1759739177 -> viable/strict/1759739177 2025-12-04T09:23:40.1757682Z * [new tag] viable/strict/1759758635 -> viable/strict/1759758635 2025-12-04T09:23:40.1757816Z * [new tag] viable/strict/1759765784 -> viable/strict/1759765784 2025-12-04T09:23:40.1757960Z * [new tag] viable/strict/1759767948 -> viable/strict/1759767948 2025-12-04T09:23:40.1758100Z * [new tag] viable/strict/1759771461 -> viable/strict/1759771461 2025-12-04T09:23:40.1758235Z * [new tag] viable/strict/1759776706 -> viable/strict/1759776706 2025-12-04T09:23:40.1758384Z * [new tag] viable/strict/1759782317 -> viable/strict/1759782317 2025-12-04T09:23:40.1758518Z * [new tag] viable/strict/1759783777 -> viable/strict/1759783777 2025-12-04T09:23:40.1758651Z * [new tag] viable/strict/1759785815 -> viable/strict/1759785815 2025-12-04T09:23:40.1759216Z * [new tag] viable/strict/1759789459 -> viable/strict/1759789459 2025-12-04T09:23:40.1760379Z * [new tag] viable/strict/1759790974 -> viable/strict/1759790974 2025-12-04T09:23:40.1760958Z * [new tag] viable/strict/1759794583 -> viable/strict/1759794583 2025-12-04T09:23:40.1766806Z * [new tag] viable/strict/1759797408 -> viable/strict/1759797408 2025-12-04T09:23:40.1766970Z * [new tag] viable/strict/1759799518 -> viable/strict/1759799518 2025-12-04T09:23:40.1767102Z * [new tag] viable/strict/1759804909 -> viable/strict/1759804909 2025-12-04T09:23:40.1767278Z * [new tag] viable/strict/1759807643 -> viable/strict/1759807643 2025-12-04T09:23:40.1767408Z * [new tag] viable/strict/1759809089 -> viable/strict/1759809089 2025-12-04T09:23:40.1767546Z * [new tag] viable/strict/1759811145 -> viable/strict/1759811145 2025-12-04T09:23:40.1767673Z * [new tag] viable/strict/1759812581 -> viable/strict/1759812581 2025-12-04T09:23:40.1767804Z * [new tag] viable/strict/1759814683 -> viable/strict/1759814683 2025-12-04T09:23:40.1767937Z * [new tag] viable/strict/1759821889 -> viable/strict/1759821889 2025-12-04T09:23:40.1770111Z * [new tag] viable/strict/1759823376 -> viable/strict/1759823376 2025-12-04T09:23:40.1770458Z * [new tag] viable/strict/1759827107 -> viable/strict/1759827107 2025-12-04T09:23:40.1770631Z * [new tag] viable/strict/1759830577 -> viable/strict/1759830577 2025-12-04T09:23:40.1770795Z * [new tag] viable/strict/1759832720 -> viable/strict/1759832720 2025-12-04T09:23:40.1770937Z * [new tag] viable/strict/1759842063 -> viable/strict/1759842063 2025-12-04T09:23:40.1771240Z * [new tag] viable/strict/1759847121 -> viable/strict/1759847121 2025-12-04T09:23:40.1771404Z * [new tag] viable/strict/1759850721 -> viable/strict/1759850721 2025-12-04T09:23:40.1776393Z * [new tag] viable/strict/1759857870 -> viable/strict/1759857870 2025-12-04T09:23:40.1776715Z * [new tag] viable/strict/1759863143 -> viable/strict/1759863143 2025-12-04T09:23:40.1777149Z * [new tag] viable/strict/1759875874 -> viable/strict/1759875874 2025-12-04T09:23:40.1777341Z * [new tag] viable/strict/1759877385 -> viable/strict/1759877385 2025-12-04T09:23:40.1777499Z * [new tag] viable/strict/1759883801 -> viable/strict/1759883801 2025-12-04T09:23:40.1777860Z * [new tag] viable/strict/1759885922 -> viable/strict/1759885922 2025-12-04T09:23:40.1778052Z * [new tag] viable/strict/1759888488 -> viable/strict/1759888488 2025-12-04T09:23:40.1778206Z * [new tag] viable/strict/1759895471 -> viable/strict/1759895471 2025-12-04T09:23:40.1778455Z * [new tag] viable/strict/1759904803 -> viable/strict/1759904803 2025-12-04T09:23:40.1778641Z * [new tag] viable/strict/1759908300 -> viable/strict/1759908300 2025-12-04T09:23:40.1778785Z * [new tag] viable/strict/1759915520 -> viable/strict/1759915520 2025-12-04T09:23:40.1778933Z * [new tag] viable/strict/1759916978 -> viable/strict/1759916978 2025-12-04T09:23:40.1779080Z * [new tag] viable/strict/1759930024 -> viable/strict/1759930024 2025-12-04T09:23:40.1779218Z * [new tag] viable/strict/1759948122 -> viable/strict/1759948122 2025-12-04T09:23:40.1779362Z * [new tag] viable/strict/1759952983 -> viable/strict/1759952983 2025-12-04T09:23:40.1779490Z * [new tag] viable/strict/1759955121 -> viable/strict/1759955121 2025-12-04T09:23:40.1779646Z * [new tag] viable/strict/1759962298 -> viable/strict/1759962298 2025-12-04T09:23:40.1785091Z * [new tag] viable/strict/1759965837 -> viable/strict/1759965837 2025-12-04T09:23:40.1785254Z * [new tag] viable/strict/1759970213 -> viable/strict/1759970213 2025-12-04T09:23:40.1785725Z * [new tag] viable/strict/1759974894 -> viable/strict/1759974894 2025-12-04T09:23:40.1785897Z * [new tag] viable/strict/1759977763 -> viable/strict/1759977763 2025-12-04T09:23:40.1786040Z * [new tag] viable/strict/1759979241 -> viable/strict/1759979241 2025-12-04T09:23:40.1786175Z * [new tag] viable/strict/1759985417 -> viable/strict/1759985417 2025-12-04T09:23:40.1786317Z * [new tag] viable/strict/1759987490 -> viable/strict/1759987490 2025-12-04T09:23:40.1787774Z * [new tag] viable/strict/1759996180 -> viable/strict/1759996180 2025-12-04T09:23:40.1787928Z * [new tag] viable/strict/1760065682 -> viable/strict/1760065682 2025-12-04T09:23:40.1788379Z * [new tag] viable/strict/1760066894 -> viable/strict/1760066894 2025-12-04T09:23:40.1788536Z * [new tag] viable/strict/1760070345 -> viable/strict/1760070345 2025-12-04T09:23:40.1788673Z * [new tag] viable/strict/1760089782 -> viable/strict/1760089782 2025-12-04T09:23:40.1788818Z * [new tag] viable/strict/1760091921 -> viable/strict/1760091921 2025-12-04T09:23:40.1788992Z * [new tag] viable/strict/1760127924 -> viable/strict/1760127924 2025-12-04T09:23:40.1789133Z * [new tag] viable/strict/1760129489 -> viable/strict/1760129489 2025-12-04T09:23:40.1789263Z * [new tag] viable/strict/1760132980 -> viable/strict/1760132980 2025-12-04T09:23:40.1789407Z * [new tag] viable/strict/1760135060 -> viable/strict/1760135060 2025-12-04T09:23:40.1794397Z * [new tag] viable/strict/1760215782 -> viable/strict/1760215782 2025-12-04T09:23:40.1794726Z * [new tag] viable/strict/1760273849 -> viable/strict/1760273849 2025-12-04T09:23:40.1794896Z * [new tag] viable/strict/1760275517 -> viable/strict/1760275517 2025-12-04T09:23:40.1795067Z * [new tag] viable/strict/1760276979 -> viable/strict/1760276979 2025-12-04T09:23:40.1795361Z * [new tag] viable/strict/1760279007 -> viable/strict/1760279007 2025-12-04T09:23:40.1795487Z * [new tag] viable/strict/1760286328 -> viable/strict/1760286328 2025-12-04T09:23:40.1795621Z * [new tag] viable/strict/1760493304 -> viable/strict/1760493304 2025-12-04T09:23:40.1795930Z * [new tag] viable/strict/1760496298 -> viable/strict/1760496298 2025-12-04T09:23:40.1796592Z * [new tag] viable/strict/1760518396 -> viable/strict/1760518396 2025-12-04T09:23:40.1796922Z * [new tag] viable/strict/1760534864 -> viable/strict/1760534864 2025-12-04T09:23:40.1797210Z * [new tag] viable/strict/1760549062 -> viable/strict/1760549062 2025-12-04T09:23:40.1797366Z * [new tag] viable/strict/1760552799 -> viable/strict/1760552799 2025-12-04T09:23:40.1797495Z * [new tag] viable/strict/1760554355 -> viable/strict/1760554355 2025-12-04T09:23:40.1797649Z * [new tag] viable/strict/1760556275 -> viable/strict/1760556275 2025-12-04T09:23:40.1797779Z * [new tag] viable/strict/1760564979 -> viable/strict/1760564979 2025-12-04T09:23:40.1797902Z * [new tag] viable/strict/1760567049 -> viable/strict/1760567049 2025-12-04T09:23:40.1798039Z * [new tag] viable/strict/1760568585 -> viable/strict/1760568585 2025-12-04T09:23:40.1798166Z * [new tag] viable/strict/1760570630 -> viable/strict/1760570630 2025-12-04T09:23:40.1798299Z * [new tag] viable/strict/1760572180 -> viable/strict/1760572180 2025-12-04T09:23:40.1800061Z * [new tag] viable/strict/1760575094 -> viable/strict/1760575094 2025-12-04T09:23:40.1800225Z * [new tag] viable/strict/1760579709 -> viable/strict/1760579709 2025-12-04T09:23:40.1800743Z * [new tag] viable/strict/1760582614 -> viable/strict/1760582614 2025-12-04T09:23:40.1801224Z * [new tag] viable/strict/1760586815 -> viable/strict/1760586815 2025-12-04T09:23:40.1801763Z * [new tag] viable/strict/1760588829 -> viable/strict/1760588829 2025-12-04T09:23:40.1802796Z * [new tag] viable/strict/1760590200 -> viable/strict/1760590200 2025-12-04T09:23:40.1803423Z * [new tag] viable/strict/1760592311 -> viable/strict/1760592311 2025-12-04T09:23:40.1803753Z * [new tag] viable/strict/1760619733 -> viable/strict/1760619733 2025-12-04T09:23:40.1804110Z * [new tag] viable/strict/1760628335 -> viable/strict/1760628335 2025-12-04T09:23:40.1810981Z * [new tag] viable/strict/1760635490 -> viable/strict/1760635490 2025-12-04T09:23:40.1816142Z * [new tag] viable/strict/1760640743 -> viable/strict/1760640743 2025-12-04T09:23:40.1821020Z * [new tag] viable/strict/1760642528 -> viable/strict/1760642528 2025-12-04T09:23:40.1825486Z * [new tag] viable/strict/1760646330 -> viable/strict/1760646330 2025-12-04T09:23:40.1831265Z * [new tag] viable/strict/1760666101 -> viable/strict/1760666101 2025-12-04T09:23:40.1831451Z * [new tag] viable/strict/1760668990 -> viable/strict/1760668990 2025-12-04T09:23:40.1831894Z * [new tag] viable/strict/1760670600 -> viable/strict/1760670600 2025-12-04T09:23:40.1832054Z * [new tag] viable/strict/1760671704 -> viable/strict/1760671704 2025-12-04T09:23:40.1832206Z * [new tag] viable/strict/1760673121 -> viable/strict/1760673121 2025-12-04T09:23:40.1832342Z * [new tag] viable/strict/1760675352 -> viable/strict/1760675352 2025-12-04T09:23:40.1832483Z * [new tag] viable/strict/1760696731 -> viable/strict/1760696731 2025-12-04T09:23:40.1832768Z * [new tag] viable/strict/1760723515 -> viable/strict/1760723515 2025-12-04T09:23:40.1832907Z * [new tag] viable/strict/1760727234 -> viable/strict/1760727234 2025-12-04T09:23:40.1833053Z * [new tag] viable/strict/1760730578 -> viable/strict/1760730578 2025-12-04T09:23:40.1833189Z * [new tag] viable/strict/1760732726 -> viable/strict/1760732726 2025-12-04T09:23:40.1833378Z * [new tag] viable/strict/1760734180 -> viable/strict/1760734180 2025-12-04T09:23:40.1833509Z * [new tag] viable/strict/1760736251 -> viable/strict/1760736251 2025-12-04T09:23:40.1833642Z * [new tag] viable/strict/1760737772 -> viable/strict/1760737772 2025-12-04T09:23:40.1833801Z * [new tag] viable/strict/1760758005 -> viable/strict/1760758005 2025-12-04T09:23:40.1833941Z * [new tag] viable/strict/1760761532 -> viable/strict/1760761532 2025-12-04T09:23:40.1834078Z * [new tag] viable/strict/1760802581 -> viable/strict/1760802581 2025-12-04T09:23:40.1834218Z * [new tag] viable/strict/1760827772 -> viable/strict/1760827772 2025-12-04T09:23:40.1834344Z * [new tag] viable/strict/1760834524 -> viable/strict/1760834524 2025-12-04T09:23:40.1834477Z * [new tag] viable/strict/1760845009 -> viable/strict/1760845009 2025-12-04T09:23:40.1834621Z * [new tag] viable/strict/1760876836 -> viable/strict/1760876836 2025-12-04T09:23:40.1834757Z * [new tag] viable/strict/1760880329 -> viable/strict/1760880329 2025-12-04T09:23:40.1834900Z * [new tag] viable/strict/1760888987 -> viable/strict/1760888987 2025-12-04T09:23:40.1835044Z * [new tag] viable/strict/1760912664 -> viable/strict/1760912664 2025-12-04T09:23:40.1835190Z * [new tag] viable/strict/1760925321 -> viable/strict/1760925321 2025-12-04T09:23:40.1835321Z * [new tag] viable/strict/1760931488 -> viable/strict/1760931488 2025-12-04T09:23:40.1835457Z * [new tag] viable/strict/1760932693 -> viable/strict/1760932693 2025-12-04T09:23:40.1835605Z * [new tag] viable/strict/1761004184 -> viable/strict/1761004184 2025-12-04T09:23:40.1835743Z * [new tag] viable/strict/1761014748 -> viable/strict/1761014748 2025-12-04T09:23:40.1835892Z * [new tag] viable/strict/1761017491 -> viable/strict/1761017491 2025-12-04T09:23:40.1836020Z * [new tag] viable/strict/1761018806 -> viable/strict/1761018806 2025-12-04T09:23:40.1836160Z * [new tag] viable/strict/1761020754 -> viable/strict/1761020754 2025-12-04T09:23:40.1836305Z * [new tag] viable/strict/1761024303 -> viable/strict/1761024303 2025-12-04T09:23:40.1836444Z * [new tag] viable/strict/1761029582 -> viable/strict/1761029582 2025-12-04T09:23:40.1836579Z * [new tag] viable/strict/1761031535 -> viable/strict/1761031535 2025-12-04T09:23:40.1836719Z * [new tag] viable/strict/1761035196 -> viable/strict/1761035196 2025-12-04T09:23:40.1836857Z * [new tag] viable/strict/1761045825 -> viable/strict/1761045825 2025-12-04T09:23:40.1836991Z * [new tag] viable/strict/1761054796 -> viable/strict/1761054796 2025-12-04T09:23:40.1837134Z * [new tag] viable/strict/1761060314 -> viable/strict/1761060314 2025-12-04T09:23:40.1837261Z * [new tag] viable/strict/1761071198 -> viable/strict/1761071198 2025-12-04T09:23:40.1837393Z * [new tag] viable/strict/1761074628 -> viable/strict/1761074628 2025-12-04T09:23:40.1837532Z * [new tag] viable/strict/1761078351 -> viable/strict/1761078351 2025-12-04T09:23:40.1837676Z * [new tag] viable/strict/1761079822 -> viable/strict/1761079822 2025-12-04T09:23:40.1837840Z * [new tag] viable/strict/1761081873 -> viable/strict/1761081873 2025-12-04T09:23:40.1837977Z * [new tag] viable/strict/1761083392 -> viable/strict/1761083392 2025-12-04T09:23:40.1838124Z * [new tag] viable/strict/1761085465 -> viable/strict/1761085465 2025-12-04T09:23:40.1838269Z * [new tag] viable/strict/1761089099 -> viable/strict/1761089099 2025-12-04T09:23:40.1838457Z * [new tag] viable/strict/1761095535 -> viable/strict/1761095535 2025-12-04T09:23:40.1838588Z * [new tag] viable/strict/1761098119 -> viable/strict/1761098119 2025-12-04T09:23:40.1838728Z * [new tag] viable/strict/1761101330 -> viable/strict/1761101330 2025-12-04T09:23:40.1838878Z * [new tag] viable/strict/1761114425 -> viable/strict/1761114425 2025-12-04T09:23:40.1839019Z * [new tag] viable/strict/1761116036 -> viable/strict/1761116036 2025-12-04T09:23:40.1839160Z * [new tag] viable/strict/1761119379 -> viable/strict/1761119379 2025-12-04T09:23:40.1839304Z * [new tag] viable/strict/1761121601 -> viable/strict/1761121601 2025-12-04T09:23:40.1839451Z * [new tag] viable/strict/1761123234 -> viable/strict/1761123234 2025-12-04T09:23:40.1840541Z * [new tag] viable/strict/1761126621 -> viable/strict/1761126621 2025-12-04T09:23:40.1841085Z * [new tag] viable/strict/1761132259 -> viable/strict/1761132259 2025-12-04T09:23:40.1841685Z * [new tag] viable/strict/1761146746 -> viable/strict/1761146746 2025-12-04T09:23:40.1842264Z * [new tag] viable/strict/1761164752 -> viable/strict/1761164752 2025-12-04T09:23:40.1842690Z * [new tag] viable/strict/1761166198 -> viable/strict/1761166198 2025-12-04T09:23:40.1844549Z * [new tag] viable/strict/1761175424 -> viable/strict/1761175424 2025-12-04T09:23:40.1844785Z * [new tag] viable/strict/1761176983 -> viable/strict/1761176983 2025-12-04T09:23:40.1844933Z * [new tag] viable/strict/1761179891 -> viable/strict/1761179891 2025-12-04T09:23:40.1845502Z * [new tag] viable/strict/1761181930 -> viable/strict/1761181930 2025-12-04T09:23:40.1846056Z * [new tag] viable/strict/1761184516 -> viable/strict/1761184516 2025-12-04T09:23:40.1846996Z * [new tag] viable/strict/1761190179 -> viable/strict/1761190179 2025-12-04T09:23:40.1847408Z * [new tag] viable/strict/1761193558 -> viable/strict/1761193558 2025-12-04T09:23:40.1848707Z * [new tag] viable/strict/1761207990 -> viable/strict/1761207990 2025-12-04T09:23:40.1848937Z * [new tag] viable/strict/1761229539 -> viable/strict/1761229539 2025-12-04T09:23:40.1850968Z * [new tag] viable/strict/1761244031 -> viable/strict/1761244031 2025-12-04T09:23:40.1851214Z * [new tag] viable/strict/1761248986 -> viable/strict/1761248986 2025-12-04T09:23:40.1851361Z * [new tag] viable/strict/1761259791 -> viable/strict/1761259791 2025-12-04T09:23:40.1851690Z * [new tag] viable/strict/1761266139 -> viable/strict/1761266139 2025-12-04T09:23:40.1852555Z * [new tag] viable/strict/1761268316 -> viable/strict/1761268316 2025-12-04T09:23:40.1852982Z * [new tag] viable/strict/1761273805 -> viable/strict/1761273805 2025-12-04T09:23:40.1856084Z * [new tag] viable/strict/1761275261 -> viable/strict/1761275261 2025-12-04T09:23:40.1856251Z * [new tag] viable/strict/1761277913 -> viable/strict/1761277913 2025-12-04T09:23:40.1856391Z * [new tag] viable/strict/1761290701 -> viable/strict/1761290701 2025-12-04T09:23:40.1856529Z * [new tag] viable/strict/1761294396 -> viable/strict/1761294396 2025-12-04T09:23:40.1856892Z * [new tag] viable/strict/1761303047 -> viable/strict/1761303047 2025-12-04T09:23:40.1857158Z * [new tag] viable/strict/1761335388 -> viable/strict/1761335388 2025-12-04T09:23:40.1857293Z * [new tag] viable/strict/1761337551 -> viable/strict/1761337551 2025-12-04T09:23:40.1859348Z * [new tag] viable/strict/1761339007 -> viable/strict/1761339007 2025-12-04T09:23:40.1859644Z * [new tag] viable/strict/1761341050 -> viable/strict/1761341050 2025-12-04T09:23:40.1859802Z * [new tag] viable/strict/1761346188 -> viable/strict/1761346188 2025-12-04T09:23:40.1860016Z * [new tag] viable/strict/1761349792 -> viable/strict/1761349792 2025-12-04T09:23:40.1861528Z * [new tag] viable/strict/1761352620 -> viable/strict/1761352620 2025-12-04T09:23:40.1861682Z * [new tag] viable/strict/1761354730 -> viable/strict/1761354730 2025-12-04T09:23:40.1861840Z * [new tag] viable/strict/1761357298 -> viable/strict/1761357298 2025-12-04T09:23:40.1862706Z * [new tag] viable/strict/1761360201 -> viable/strict/1761360201 2025-12-04T09:23:40.1863373Z * [new tag] viable/strict/1761361753 -> viable/strict/1761361753 2025-12-04T09:23:40.1863757Z * [new tag] viable/strict/1761364351 -> viable/strict/1761364351 2025-12-04T09:23:40.1864215Z * [new tag] viable/strict/1761366338 -> viable/strict/1761366338 2025-12-04T09:23:40.1865208Z * [new tag] viable/strict/1761367802 -> viable/strict/1761367802 2025-12-04T09:23:40.1865444Z * [new tag] viable/strict/1761369889 -> viable/strict/1761369889 2025-12-04T09:23:40.1868488Z * [new tag] viable/strict/1761371385 -> viable/strict/1761371385 2025-12-04T09:23:40.1868652Z * [new tag] viable/strict/1761373581 -> viable/strict/1761373581 2025-12-04T09:23:40.1868900Z * [new tag] viable/strict/1761375054 -> viable/strict/1761375054 2025-12-04T09:23:40.1869165Z * [new tag] viable/strict/1761421785 -> viable/strict/1761421785 2025-12-04T09:23:40.1869290Z * [new tag] viable/strict/1761434614 -> viable/strict/1761434614 2025-12-04T09:23:40.1873806Z * [new tag] viable/strict/1761439254 -> viable/strict/1761439254 2025-12-04T09:23:40.1875681Z * [new tag] viable/strict/1761454187 -> viable/strict/1761454187 2025-12-04T09:23:40.1875824Z * [new tag] viable/strict/1761459991 -> viable/strict/1761459991 2025-12-04T09:23:40.1875966Z * [new tag] viable/strict/1761470668 -> viable/strict/1761470668 2025-12-04T09:23:40.1876102Z * [new tag] viable/strict/1761472188 -> viable/strict/1761472188 2025-12-04T09:23:40.1876238Z * [new tag] viable/strict/1761503178 -> viable/strict/1761503178 2025-12-04T09:23:40.1876381Z * [new tag] viable/strict/1761517492 -> viable/strict/1761517492 2025-12-04T09:23:40.1876514Z * [new tag] viable/strict/1761518981 -> viable/strict/1761518981 2025-12-04T09:23:40.1876649Z * [new tag] viable/strict/1761533609 -> viable/strict/1761533609 2025-12-04T09:23:40.1876783Z * [new tag] viable/strict/1761546438 -> viable/strict/1761546438 2025-12-04T09:23:40.1877536Z * [new tag] viable/strict/1761548133 -> viable/strict/1761548133 2025-12-04T09:23:40.1877873Z * [new tag] viable/strict/1761555186 -> viable/strict/1761555186 2025-12-04T09:23:40.1878464Z * [new tag] viable/strict/1761557178 -> viable/strict/1761557178 2025-12-04T09:23:40.1878978Z * [new tag] viable/strict/1761560772 -> viable/strict/1761560772 2025-12-04T09:23:40.1880001Z * [new tag] viable/strict/1761562266 -> viable/strict/1761562266 2025-12-04T09:23:40.1880370Z * [new tag] viable/strict/1761564260 -> viable/strict/1761564260 2025-12-04T09:23:40.1881675Z * [new tag] viable/strict/1761568072 -> viable/strict/1761568072 2025-12-04T09:23:40.1881825Z * [new tag] viable/strict/1761571683 -> viable/strict/1761571683 2025-12-04T09:23:40.1882245Z * [new tag] viable/strict/1761580199 -> viable/strict/1761580199 2025-12-04T09:23:40.1882917Z * [new tag] viable/strict/1761587383 -> viable/strict/1761587383 2025-12-04T09:23:40.1883783Z * [new tag] viable/strict/1761591165 -> viable/strict/1761591165 2025-12-04T09:23:40.1884399Z * [new tag] viable/strict/1761594575 -> viable/strict/1761594575 2025-12-04T09:23:40.1884760Z * [new tag] viable/strict/1761596710 -> viable/strict/1761596710 2025-12-04T09:23:40.1885282Z * [new tag] viable/strict/1761598189 -> viable/strict/1761598189 2025-12-04T09:23:40.1887679Z * [new tag] viable/strict/1761600254 -> viable/strict/1761600254 2025-12-04T09:23:40.1888240Z * [new tag] viable/strict/1761603879 -> viable/strict/1761603879 2025-12-04T09:23:40.1888485Z * [new tag] viable/strict/1761605429 -> viable/strict/1761605429 2025-12-04T09:23:40.1888674Z * [new tag] viable/strict/1761607468 -> viable/strict/1761607468 2025-12-04T09:23:40.1888821Z * [new tag] viable/strict/1761608983 -> viable/strict/1761608983 2025-12-04T09:23:40.1889339Z * [new tag] viable/strict/1761611846 -> viable/strict/1761611846 2025-12-04T09:23:40.1889871Z * [new tag] viable/strict/1761613922 -> viable/strict/1761613922 2025-12-04T09:23:40.1890383Z * [new tag] viable/strict/1761616504 -> viable/strict/1761616504 2025-12-04T09:23:40.1890834Z * [new tag] viable/strict/1761619599 -> viable/strict/1761619599 2025-12-04T09:23:40.1891691Z * [new tag] viable/strict/1761686693 -> viable/strict/1761686693 2025-12-04T09:23:40.1892072Z * [new tag] viable/strict/1761688179 -> viable/strict/1761688179 2025-12-04T09:23:40.1894945Z * [new tag] viable/strict/1761691973 -> viable/strict/1761691973 2025-12-04T09:23:40.1895149Z * [new tag] viable/strict/1761693884 -> viable/strict/1761693884 2025-12-04T09:23:40.1895287Z * [new tag] viable/strict/1761695389 -> viable/strict/1761695389 2025-12-04T09:23:40.1895418Z * [new tag] viable/strict/1761698408 -> viable/strict/1761698408 2025-12-04T09:23:40.1895565Z * [new tag] viable/strict/1761702931 -> viable/strict/1761702931 2025-12-04T09:23:40.1896009Z * [new tag] viable/strict/1761706307 -> viable/strict/1761706307 2025-12-04T09:23:40.1896667Z * [new tag] viable/strict/1761709065 -> viable/strict/1761709065 2025-12-04T09:23:40.1897564Z * [new tag] viable/strict/1761710285 -> viable/strict/1761710285 2025-12-04T09:23:40.1897847Z * [new tag] viable/strict/1761711983 -> viable/strict/1761711983 2025-12-04T09:23:40.1902310Z * [new tag] viable/strict/1761713514 -> viable/strict/1761713514 2025-12-04T09:23:40.1902493Z * [new tag] viable/strict/1761715523 -> viable/strict/1761715523 2025-12-04T09:23:40.1902639Z * [new tag] viable/strict/1761727973 -> viable/strict/1761727973 2025-12-04T09:23:40.1902768Z * [new tag] viable/strict/1761751558 -> viable/strict/1761751558 2025-12-04T09:23:40.1902905Z * [new tag] viable/strict/1761755187 -> viable/strict/1761755187 2025-12-04T09:23:40.1903034Z * [new tag] viable/strict/1761756826 -> viable/strict/1761756826 2025-12-04T09:23:40.1903166Z * [new tag] viable/strict/1761769551 -> viable/strict/1761769551 2025-12-04T09:23:40.1903513Z * [new tag] viable/strict/1761771032 -> viable/strict/1761771032 2025-12-04T09:23:40.1903923Z * [new tag] viable/strict/1761773101 -> viable/strict/1761773101 2025-12-04T09:23:40.1904775Z * [new tag] viable/strict/1761781792 -> viable/strict/1761781792 2025-12-04T09:23:40.1905233Z * [new tag] viable/strict/1761784788 -> viable/strict/1761784788 2025-12-04T09:23:40.1906154Z * [new tag] viable/strict/1761786740 -> viable/strict/1761786740 2025-12-04T09:23:40.1906803Z * [new tag] viable/strict/1761789332 -> viable/strict/1761789332 2025-12-04T09:23:40.1907692Z * [new tag] viable/strict/1761792569 -> viable/strict/1761792569 2025-12-04T09:23:40.1908117Z * [new tag] viable/strict/1761795289 -> viable/strict/1761795289 2025-12-04T09:23:40.1910370Z * [new tag] viable/strict/1761798345 -> viable/strict/1761798345 2025-12-04T09:23:40.1910546Z * [new tag] viable/strict/1761799827 -> viable/strict/1761799827 2025-12-04T09:23:40.1910681Z * [new tag] viable/strict/1761805604 -> viable/strict/1761805604 2025-12-04T09:23:40.1910801Z * [new tag] viable/strict/1761807202 -> viable/strict/1761807202 2025-12-04T09:23:40.1911370Z * [new tag] viable/strict/1761809094 -> viable/strict/1761809094 2025-12-04T09:23:40.1911848Z * [new tag] viable/strict/1761810576 -> viable/strict/1761810576 2025-12-04T09:23:40.1912672Z * [new tag] viable/strict/1761812771 -> viable/strict/1761812771 2025-12-04T09:23:40.1913056Z * [new tag] viable/strict/1761814363 -> viable/strict/1761814363 2025-12-04T09:23:40.1913954Z * [new tag] viable/strict/1761857410 -> viable/strict/1761857410 2025-12-04T09:23:40.1914263Z * [new tag] viable/strict/1761860985 -> viable/strict/1761860985 2025-12-04T09:23:40.1917060Z * [new tag] viable/strict/1761863094 -> viable/strict/1761863094 2025-12-04T09:23:40.1917225Z * [new tag] viable/strict/1761864590 -> viable/strict/1761864590 2025-12-04T09:23:40.1917366Z * [new tag] viable/strict/1761866675 -> viable/strict/1761866675 2025-12-04T09:23:40.1917514Z * [new tag] viable/strict/1761868178 -> viable/strict/1761868178 2025-12-04T09:23:40.1918135Z * [new tag] viable/strict/1761871111 -> viable/strict/1761871111 2025-12-04T09:23:40.1919412Z * [new tag] viable/strict/1761873126 -> viable/strict/1761873126 2025-12-04T09:23:40.1919659Z * [new tag] viable/strict/1761875714 -> viable/strict/1761875714 2025-12-04T09:23:40.1920113Z * [new tag] viable/strict/1761878924 -> viable/strict/1761878924 2025-12-04T09:23:40.1920827Z * [new tag] viable/strict/1761881727 -> viable/strict/1761881727 2025-12-04T09:23:40.1921719Z * [new tag] viable/strict/1761882959 -> viable/strict/1761882959 2025-12-04T09:23:40.1922120Z * [new tag] viable/strict/1761886268 -> viable/strict/1761886268 2025-12-04T09:23:40.1923055Z * [new tag] viable/strict/1761893641 -> viable/strict/1761893641 2025-12-04T09:23:40.1923400Z * [new tag] viable/strict/1761931517 -> viable/strict/1761931517 2025-12-04T09:23:40.1925877Z * [new tag] viable/strict/1761933080 -> viable/strict/1761933080 2025-12-04T09:23:40.1926057Z * [new tag] viable/strict/1761935217 -> viable/strict/1761935217 2025-12-04T09:23:40.1926192Z * [new tag] viable/strict/1761938533 -> viable/strict/1761938533 2025-12-04T09:23:40.1926323Z * [new tag] viable/strict/1761940184 -> viable/strict/1761940184 2025-12-04T09:23:40.1926901Z * [new tag] viable/strict/1761942338 -> viable/strict/1761942338 2025-12-04T09:23:40.1933265Z * [new tag] viable/strict/1761946100 -> viable/strict/1761946100 2025-12-04T09:23:40.1938066Z * [new tag] viable/strict/1761947374 -> viable/strict/1761947374 2025-12-04T09:23:40.1943109Z * [new tag] viable/strict/1761950978 -> viable/strict/1761950978 2025-12-04T09:23:40.1947934Z * [new tag] viable/strict/1761957727 -> viable/strict/1761957727 2025-12-04T09:23:40.1952918Z * [new tag] viable/strict/1761959532 -> viable/strict/1761959532 2025-12-04T09:23:40.1958463Z * [new tag] viable/strict/1761965366 -> viable/strict/1761965366 2025-12-04T09:23:40.1958668Z * [new tag] viable/strict/1761968066 -> viable/strict/1761968066 2025-12-04T09:23:40.1958804Z * [new tag] viable/strict/1761969322 -> viable/strict/1761969322 2025-12-04T09:23:40.1958945Z * [new tag] viable/strict/1761974723 -> viable/strict/1761974723 2025-12-04T09:23:40.1959090Z * [new tag] viable/strict/1761981837 -> viable/strict/1761981837 2025-12-04T09:23:40.1959227Z * [new tag] viable/strict/1761985546 -> viable/strict/1761985546 2025-12-04T09:23:40.1959358Z * [new tag] viable/strict/1761987030 -> viable/strict/1761987030 2025-12-04T09:23:40.1959491Z * [new tag] viable/strict/1762003554 -> viable/strict/1762003554 2025-12-04T09:23:40.1959627Z * [new tag] viable/strict/1762021560 -> viable/strict/1762021560 2025-12-04T09:23:40.1959755Z * [new tag] viable/strict/1762032190 -> viable/strict/1762032190 2025-12-04T09:23:40.1959880Z * [new tag] viable/strict/1762040981 -> viable/strict/1762040981 2025-12-04T09:23:40.1960014Z * [new tag] viable/strict/1762048525 -> viable/strict/1762048525 2025-12-04T09:23:40.1960139Z * [new tag] viable/strict/1762104223 -> viable/strict/1762104223 2025-12-04T09:23:40.1960274Z * [new tag] viable/strict/1762105778 -> viable/strict/1762105778 2025-12-04T09:23:40.1960614Z * [new tag] viable/strict/1762115109 -> viable/strict/1762115109 2025-12-04T09:23:40.1960748Z * [new tag] viable/strict/1762125840 -> viable/strict/1762125840 2025-12-04T09:23:40.1960893Z * [new tag] viable/strict/1762127377 -> viable/strict/1762127377 2025-12-04T09:23:40.1961024Z * [new tag] viable/strict/1762134925 -> viable/strict/1762134925 2025-12-04T09:23:40.1961166Z * [new tag] viable/strict/1762138338 -> viable/strict/1762138338 2025-12-04T09:23:40.1961296Z * [new tag] viable/strict/1762148993 -> viable/strict/1762148993 2025-12-04T09:23:40.1961424Z * [new tag] viable/strict/1762152871 -> viable/strict/1762152871 2025-12-04T09:23:40.1961564Z * [new tag] viable/strict/1762156183 -> viable/strict/1762156183 2025-12-04T09:23:40.1961696Z * [new tag] viable/strict/1762163457 -> viable/strict/1762163457 2025-12-04T09:23:40.1961829Z * [new tag] viable/strict/1762165569 -> viable/strict/1762165569 2025-12-04T09:23:40.1961957Z * [new tag] viable/strict/1762169035 -> viable/strict/1762169035 2025-12-04T09:23:40.1962078Z * [new tag] viable/strict/1762174936 -> viable/strict/1762174936 2025-12-04T09:23:40.1962205Z * [new tag] viable/strict/1762194412 -> viable/strict/1762194412 2025-12-04T09:23:40.1962323Z * [new tag] viable/strict/1762195876 -> viable/strict/1762195876 2025-12-04T09:23:40.1962440Z * [new tag] viable/strict/1762197788 -> viable/strict/1762197788 2025-12-04T09:23:40.1962566Z * [new tag] viable/strict/1762199389 -> viable/strict/1762199389 2025-12-04T09:23:40.1962682Z * [new tag] viable/strict/1762206585 -> viable/strict/1762206585 2025-12-04T09:23:40.1963004Z * [new tag] viable/strict/1762210184 -> viable/strict/1762210184 2025-12-04T09:23:40.1963125Z * [new tag] viable/strict/1762218736 -> viable/strict/1762218736 2025-12-04T09:23:40.1963246Z * [new tag] viable/strict/1762224529 -> viable/strict/1762224529 2025-12-04T09:23:40.1963422Z * [new tag] viable/strict/1762227253 -> viable/strict/1762227253 2025-12-04T09:23:40.1963543Z * [new tag] viable/strict/1762228515 -> viable/strict/1762228515 2025-12-04T09:23:40.1963661Z * [new tag] viable/strict/1762230349 -> viable/strict/1762230349 2025-12-04T09:23:40.1963790Z * [new tag] viable/strict/1762231859 -> viable/strict/1762231859 2025-12-04T09:23:40.1963909Z * [new tag] viable/strict/1762233925 -> viable/strict/1762233925 2025-12-04T09:23:40.1964034Z * [new tag] viable/strict/1762237630 -> viable/strict/1762237630 2025-12-04T09:23:40.1964155Z * [new tag] viable/strict/1762253522 -> viable/strict/1762253522 2025-12-04T09:23:40.1964275Z * [new tag] viable/strict/1762278588 -> viable/strict/1762278588 2025-12-04T09:23:40.1964409Z * [new tag] viable/strict/1762284203 -> viable/strict/1762284203 2025-12-04T09:23:40.1964580Z * [new tag] viable/strict/1762289446 -> viable/strict/1762289446 2025-12-04T09:23:40.1964733Z * [new tag] viable/strict/1762291515 -> viable/strict/1762291515 2025-12-04T09:23:40.1964852Z * [new tag] viable/strict/1762295100 -> viable/strict/1762295100 2025-12-04T09:23:40.1964971Z * [new tag] viable/strict/1762296590 -> viable/strict/1762296590 2025-12-04T09:23:40.1965099Z * [new tag] viable/strict/1762300179 -> viable/strict/1762300179 2025-12-04T09:23:40.1965218Z * [new tag] viable/strict/1762303207 -> viable/strict/1762303207 2025-12-04T09:23:40.1965346Z * [new tag] viable/strict/1762386584 -> viable/strict/1762386584 2025-12-04T09:23:40.1965566Z * [new tag] viable/strict/1762391537 -> viable/strict/1762391537 2025-12-04T09:23:40.1972108Z * [new tag] viable/strict/1762394119 -> viable/strict/1762394119 2025-12-04T09:23:40.1974063Z * [new tag] viable/strict/1762397437 -> viable/strict/1762397437 2025-12-04T09:23:40.1974335Z * [new tag] viable/strict/1762400256 -> viable/strict/1762400256 2025-12-04T09:23:40.1978054Z * [new tag] viable/strict/1762401469 -> viable/strict/1762401469 2025-12-04T09:23:40.1982984Z * [new tag] viable/strict/1762408195 -> viable/strict/1762408195 2025-12-04T09:23:40.1988467Z * [new tag] viable/strict/1762410411 -> viable/strict/1762410411 2025-12-04T09:23:40.1990344Z * [new tag] viable/strict/1762417613 -> viable/strict/1762417613 2025-12-04T09:23:40.1990645Z * [new tag] viable/strict/1762419198 -> viable/strict/1762419198 2025-12-04T09:23:40.1990808Z * [new tag] viable/strict/1762422656 -> viable/strict/1762422656 2025-12-04T09:23:40.1990924Z * [new tag] viable/strict/1762424746 -> viable/strict/1762424746 2025-12-04T09:23:40.1991047Z * [new tag] viable/strict/1762446386 -> viable/strict/1762446386 2025-12-04T09:23:40.1991160Z * [new tag] viable/strict/1762449912 -> viable/strict/1762449912 2025-12-04T09:23:40.1991273Z * [new tag] viable/strict/1762457031 -> viable/strict/1762457031 2025-12-04T09:23:40.1991390Z * [new tag] viable/strict/1762462441 -> viable/strict/1762462441 2025-12-04T09:23:40.1991503Z * [new tag] viable/strict/1762467909 -> viable/strict/1762467909 2025-12-04T09:23:40.1991616Z * [new tag] viable/strict/1762471493 -> viable/strict/1762471493 2025-12-04T09:23:40.1991868Z * [new tag] viable/strict/1762475990 -> viable/strict/1762475990 2025-12-04T09:23:40.1991983Z * [new tag] viable/strict/1762477933 -> viable/strict/1762477933 2025-12-04T09:23:40.1992106Z * [new tag] viable/strict/1762491053 -> viable/strict/1762491053 2025-12-04T09:23:40.1992261Z * [new tag] viable/strict/1762493118 -> viable/strict/1762493118 2025-12-04T09:23:40.1992377Z * [new tag] viable/strict/1762498442 -> viable/strict/1762498442 2025-12-04T09:23:40.1992502Z * [new tag] viable/strict/1762501778 -> viable/strict/1762501778 2025-12-04T09:23:40.1992615Z * [new tag] viable/strict/1762504001 -> viable/strict/1762504001 2025-12-04T09:23:40.1992841Z * [new tag] viable/strict/1762505583 -> viable/strict/1762505583 2025-12-04T09:23:40.1992957Z * [new tag] viable/strict/1762507523 -> viable/strict/1762507523 2025-12-04T09:23:40.1996951Z * [new tag] viable/strict/1762511140 -> viable/strict/1762511140 2025-12-04T09:23:40.1997137Z * [new tag] viable/strict/1762512632 -> viable/strict/1762512632 2025-12-04T09:23:40.1997273Z * [new tag] viable/strict/1762520467 -> viable/strict/1762520467 2025-12-04T09:23:40.1997426Z * [new tag] viable/strict/1762522016 -> viable/strict/1762522016 2025-12-04T09:23:40.1997559Z * [new tag] viable/strict/1762530591 -> viable/strict/1762530591 2025-12-04T09:23:40.1997691Z * [new tag] viable/strict/1762543405 -> viable/strict/1762543405 2025-12-04T09:23:40.1997829Z * [new tag] viable/strict/1762544998 -> viable/strict/1762544998 2025-12-04T09:23:40.1997956Z * [new tag] viable/strict/1762552182 -> viable/strict/1762552182 2025-12-04T09:23:40.1998091Z * [new tag] viable/strict/1762554297 -> viable/strict/1762554297 2025-12-04T09:23:40.1998225Z * [new tag] viable/strict/1762559381 -> viable/strict/1762559381 2025-12-04T09:23:40.1998352Z * [new tag] viable/strict/1762562222 -> viable/strict/1762562222 2025-12-04T09:23:40.1998486Z * [new tag] viable/strict/1762564319 -> viable/strict/1762564319 2025-12-04T09:23:40.1998616Z * [new tag] viable/strict/1762566904 -> viable/strict/1762566904 2025-12-04T09:23:40.1998749Z * [new tag] viable/strict/1762569781 -> viable/strict/1762569781 2025-12-04T09:23:40.1998873Z * [new tag] viable/strict/1762575940 -> viable/strict/1762575940 2025-12-04T09:23:40.1999053Z * [new tag] viable/strict/1762580974 -> viable/strict/1762580974 2025-12-04T09:23:40.1999192Z * [new tag] viable/strict/1762583185 -> viable/strict/1762583185 2025-12-04T09:23:40.1999318Z * [new tag] viable/strict/1762586647 -> viable/strict/1762586647 2025-12-04T09:23:40.1999449Z * [new tag] viable/strict/1762588183 -> viable/strict/1762588183 2025-12-04T09:23:40.1999585Z * [new tag] viable/strict/1762593886 -> viable/strict/1762593886 2025-12-04T09:23:40.1999713Z * [new tag] viable/strict/1762650743 -> viable/strict/1762650743 2025-12-04T09:23:40.1999850Z * [new tag] viable/strict/1762653328 -> viable/strict/1762653328 2025-12-04T09:23:40.1999981Z * [new tag] viable/strict/1762659342 -> viable/strict/1762659342 2025-12-04T09:23:40.2000130Z * [new tag] viable/strict/1762662360 -> viable/strict/1762662360 2025-12-04T09:23:40.2000302Z * [new tag] viable/strict/1762667377 -> viable/strict/1762667377 2025-12-04T09:23:40.2000523Z * [new tag] viable/strict/1762671090 -> viable/strict/1762671090 2025-12-04T09:23:40.2000661Z * [new tag] viable/strict/1762680284 -> viable/strict/1762680284 2025-12-04T09:23:40.2000940Z * [new tag] viable/strict/1762683900 -> viable/strict/1762683900 2025-12-04T09:23:40.2001077Z * [new tag] viable/strict/1762705541 -> viable/strict/1762705541 2025-12-04T09:23:40.2001223Z * [new tag] viable/strict/1762709004 -> viable/strict/1762709004 2025-12-04T09:23:40.2001488Z * [new tag] viable/strict/1762746004 -> viable/strict/1762746004 2025-12-04T09:23:40.2001641Z * [new tag] viable/strict/1762748799 -> viable/strict/1762748799 2025-12-04T09:23:40.2005806Z * [new tag] viable/strict/1762759504 -> viable/strict/1762759504 2025-12-04T09:23:40.2011289Z * [new tag] viable/strict/1762760973 -> viable/strict/1762760973 2025-12-04T09:23:40.2014869Z * [new tag] viable/strict/1762775374 -> viable/strict/1762775374 2025-12-04T09:23:40.2019863Z * [new tag] viable/strict/1762777661 -> viable/strict/1762777661 2025-12-04T09:23:40.2020052Z * [new tag] viable/strict/1762779774 -> viable/strict/1762779774 2025-12-04T09:23:40.2020173Z * [new tag] viable/strict/1762781259 -> viable/strict/1762781259 2025-12-04T09:23:40.2020299Z * [new tag] viable/strict/1762793628 -> viable/strict/1762793628 2025-12-04T09:23:40.2020422Z * [new tag] viable/strict/1762800711 -> viable/strict/1762800711 2025-12-04T09:23:40.2020543Z * [new tag] viable/strict/1762809894 -> viable/strict/1762809894 2025-12-04T09:23:40.2020668Z * [new tag] viable/strict/1762811384 -> viable/strict/1762811384 2025-12-04T09:23:40.2020783Z * [new tag] viable/strict/1762813841 -> viable/strict/1762813841 2025-12-04T09:23:40.2020905Z * [new tag] viable/strict/1762815047 -> viable/strict/1762815047 2025-12-04T09:23:40.2021025Z * [new tag] viable/strict/1762817094 -> viable/strict/1762817094 2025-12-04T09:23:40.2021145Z * [new tag] viable/strict/1762818582 -> viable/strict/1762818582 2025-12-04T09:23:40.2021274Z * [new tag] viable/strict/1762821623 -> viable/strict/1762821623 2025-12-04T09:23:40.2021391Z * [new tag] viable/strict/1762823531 -> viable/strict/1762823531 2025-12-04T09:23:40.2021512Z * [new tag] viable/strict/1762849583 -> viable/strict/1762849583 2025-12-04T09:23:40.2021638Z * [new tag] viable/strict/1762851200 -> viable/strict/1762851200 2025-12-04T09:23:40.2021755Z * [new tag] viable/strict/1762854603 -> viable/strict/1762854603 2025-12-04T09:23:40.2021878Z * [new tag] viable/strict/1762858276 -> viable/strict/1762858276 2025-12-04T09:23:40.2021995Z * [new tag] viable/strict/1762860891 -> viable/strict/1762860891 2025-12-04T09:23:40.2022113Z * [new tag] viable/strict/1762866174 -> viable/strict/1762866174 2025-12-04T09:23:40.2022240Z * [new tag] viable/strict/1762867653 -> viable/strict/1762867653 2025-12-04T09:23:40.2022357Z * [new tag] viable/strict/1762872669 -> viable/strict/1762872669 2025-12-04T09:23:40.2022481Z * [new tag] viable/strict/1762878380 -> viable/strict/1762878380 2025-12-04T09:23:40.2022601Z * [new tag] viable/strict/1762889003 -> viable/strict/1762889003 2025-12-04T09:23:40.2022718Z * [new tag] viable/strict/1762890589 -> viable/strict/1762890589 2025-12-04T09:23:40.2022842Z * [new tag] viable/strict/1762892743 -> viable/strict/1762892743 2025-12-04T09:23:40.2022956Z * [new tag] viable/strict/1762894271 -> viable/strict/1762894271 2025-12-04T09:23:40.2023072Z * [new tag] viable/strict/1762896287 -> viable/strict/1762896287 2025-12-04T09:23:40.2023240Z * [new tag] viable/strict/1762915871 -> viable/strict/1762915871 2025-12-04T09:23:40.2023605Z * [new tag] viable/strict/1762918569 -> viable/strict/1762918569 2025-12-04T09:23:40.2023747Z * [new tag] viable/strict/1762919776 -> viable/strict/1762919776 2025-12-04T09:23:40.2023950Z * [new tag] viable/strict/1762923072 -> viable/strict/1762923072 2025-12-04T09:23:40.2024266Z * [new tag] viable/strict/1762928826 -> viable/strict/1762928826 2025-12-04T09:23:40.2024467Z * [new tag] viable/strict/1762930451 -> viable/strict/1762930451 2025-12-04T09:23:40.2029262Z * [new tag] viable/strict/1762933780 -> viable/strict/1762933780 2025-12-04T09:23:40.2029510Z * [new tag] viable/strict/1762937638 -> viable/strict/1762937638 2025-12-04T09:23:40.2034589Z * [new tag] viable/strict/1762939545 -> viable/strict/1762939545 2025-12-04T09:23:40.2037974Z * [new tag] viable/strict/1762962692 -> viable/strict/1762962692 2025-12-04T09:23:40.2038159Z * [new tag] viable/strict/1762979143 -> viable/strict/1762979143 2025-12-04T09:23:40.2038290Z * [new tag] viable/strict/1762984188 -> viable/strict/1762984188 2025-12-04T09:23:40.2038420Z * [new tag] viable/strict/1762986306 -> viable/strict/1762986306 2025-12-04T09:23:40.2038553Z * [new tag] viable/strict/1762989903 -> viable/strict/1762989903 2025-12-04T09:23:40.2038688Z * [new tag] viable/strict/1762991377 -> viable/strict/1762991377 2025-12-04T09:23:40.2038813Z * [new tag] viable/strict/1762998921 -> viable/strict/1762998921 2025-12-04T09:23:40.2038937Z * [new tag] viable/strict/1763002287 -> viable/strict/1763002287 2025-12-04T09:23:40.2039070Z * [new tag] viable/strict/1763016840 -> viable/strict/1763016840 2025-12-04T09:23:40.2039194Z * [new tag] viable/strict/1763020180 -> viable/strict/1763020180 2025-12-04T09:23:40.2039329Z * [new tag] viable/strict/1763027421 -> viable/strict/1763027421 2025-12-04T09:23:40.2039454Z * [new tag] viable/strict/1763031120 -> viable/strict/1763031120 2025-12-04T09:23:40.2039581Z * [new tag] viable/strict/1763036861 -> viable/strict/1763036861 2025-12-04T09:23:40.2039714Z * [new tag] viable/strict/1763038993 -> viable/strict/1763038993 2025-12-04T09:23:40.2039845Z * [new tag] viable/strict/1763054703 -> viable/strict/1763054703 2025-12-04T09:23:40.2039970Z * [new tag] viable/strict/1763067061 -> viable/strict/1763067061 2025-12-04T09:23:40.2040116Z * [new tag] viable/strict/1763070847 -> viable/strict/1763070847 2025-12-04T09:23:40.2040248Z * [new tag] viable/strict/1763072706 -> viable/strict/1763072706 2025-12-04T09:23:40.2040645Z * [new tag] viable/strict/1763076302 -> viable/strict/1763076302 2025-12-04T09:23:40.2040805Z * [new tag] viable/strict/1763080816 -> viable/strict/1763080816 2025-12-04T09:23:40.2040938Z * [new tag] viable/strict/1763082732 -> viable/strict/1763082732 2025-12-04T09:23:40.2041075Z * [new tag] viable/strict/1763085329 -> viable/strict/1763085329 2025-12-04T09:23:40.2041212Z * [new tag] viable/strict/1763088623 -> viable/strict/1763088623 2025-12-04T09:23:40.2044182Z * [new tag] viable/strict/1763091402 -> viable/strict/1763091402 2025-12-04T09:23:40.2049119Z * [new tag] viable/strict/1763092602 -> viable/strict/1763092602 2025-12-04T09:23:40.2054309Z * [new tag] viable/strict/1763094355 -> viable/strict/1763094355 2025-12-04T09:23:40.2054626Z * [new tag] viable/strict/1763099390 -> viable/strict/1763099390 2025-12-04T09:23:40.2054807Z * [new tag] viable/strict/1763101608 -> viable/strict/1763101608 2025-12-04T09:23:40.2055294Z * [new tag] viable/strict/1763105102 -> viable/strict/1763105102 2025-12-04T09:23:40.2055456Z * [new tag] viable/strict/1763112347 -> viable/strict/1763112347 2025-12-04T09:23:40.2056196Z * [new tag] viable/strict/1763119471 -> viable/strict/1763119471 2025-12-04T09:23:40.2056651Z * [new tag] viable/strict/1763126835 -> viable/strict/1763126835 2025-12-04T09:23:40.2057078Z * [new tag] viable/strict/1763149779 -> viable/strict/1763149779 2025-12-04T09:23:40.2057224Z * [new tag] viable/strict/1763164178 -> viable/strict/1763164178 2025-12-04T09:23:40.2057363Z * [new tag] viable/strict/1763167104 -> viable/strict/1763167104 2025-12-04T09:23:40.2057508Z * [new tag] viable/strict/1763169132 -> viable/strict/1763169132 2025-12-04T09:23:40.2057648Z * [new tag] viable/strict/1763171708 -> viable/strict/1763171708 2025-12-04T09:23:40.2057799Z * [new tag] viable/strict/1763174759 -> viable/strict/1763174759 2025-12-04T09:23:40.2057950Z * [new tag] viable/strict/1763180744 -> viable/strict/1763180744 2025-12-04T09:23:40.2058094Z * [new tag] viable/strict/1763182227 -> viable/strict/1763182227 2025-12-04T09:23:40.2058247Z * [new tag] viable/strict/1763184309 -> viable/strict/1763184309 2025-12-04T09:23:40.2058555Z * [new tag] viable/strict/1763187991 -> viable/strict/1763187991 2025-12-04T09:23:40.2058712Z * [new tag] viable/strict/1763191445 -> viable/strict/1763191445 2025-12-04T09:23:40.2058981Z * [new tag] viable/strict/1763195152 -> viable/strict/1763195152 2025-12-04T09:23:40.2059123Z * [new tag] viable/strict/1763205769 -> viable/strict/1763205769 2025-12-04T09:23:40.2059358Z * [new tag] viable/strict/1763246990 -> viable/strict/1763246990 2025-12-04T09:23:40.2059499Z * [new tag] viable/strict/1763261578 -> viable/strict/1763261578 2025-12-04T09:23:40.2059985Z * [new tag] viable/strict/1763286573 -> viable/strict/1763286573 2025-12-04T09:23:40.2060131Z * [new tag] viable/strict/1763292167 -> viable/strict/1763292167 2025-12-04T09:23:40.2060389Z * [new tag] viable/strict/1763333386 -> viable/strict/1763333386 2025-12-04T09:23:40.2060534Z * [new tag] viable/strict/1763340082 -> viable/strict/1763340082 2025-12-04T09:23:40.2060657Z * [new tag] viable/strict/1763364324 -> viable/strict/1763364324 2025-12-04T09:23:40.2060933Z * [new tag] viable/strict/1763371569 -> viable/strict/1763371569 2025-12-04T09:23:40.2066056Z * [new tag] viable/strict/1763373067 -> viable/strict/1763373067 2025-12-04T09:23:40.2066376Z * [new tag] viable/strict/1763375157 -> viable/strict/1763375157 2025-12-04T09:23:40.2066575Z * [new tag] viable/strict/1763382462 -> viable/strict/1763382462 2025-12-04T09:23:40.2066719Z * [new tag] viable/strict/1763394661 -> viable/strict/1763394661 2025-12-04T09:23:40.2066944Z * [new tag] viable/strict/1763396797 -> viable/strict/1763396797 2025-12-04T09:23:40.2067248Z * [new tag] viable/strict/1763398542 -> viable/strict/1763398542 2025-12-04T09:23:40.2067411Z * [new tag] viable/strict/1763401807 -> viable/strict/1763401807 2025-12-04T09:23:40.2068045Z * [new tag] viable/strict/1763414698 -> viable/strict/1763414698 2025-12-04T09:23:40.2068371Z * [new tag] viable/strict/1763419807 -> viable/strict/1763419807 2025-12-04T09:23:40.2068517Z * [new tag] viable/strict/1763426369 -> viable/strict/1763426369 2025-12-04T09:23:40.2068648Z * [new tag] viable/strict/1763428331 -> viable/strict/1763428331 2025-12-04T09:23:40.2069068Z * [new tag] viable/strict/1763430922 -> viable/strict/1763430922 2025-12-04T09:23:40.2069219Z * [new tag] viable/strict/1763434184 -> viable/strict/1763434184 2025-12-04T09:23:40.2074985Z * [new tag] viable/strict/1763439973 -> viable/strict/1763439973 2025-12-04T09:23:40.2075322Z * [new tag] viable/strict/1763444995 -> viable/strict/1763444995 2025-12-04T09:23:40.2075492Z * [new tag] viable/strict/1763447206 -> viable/strict/1763447206 2025-12-04T09:23:40.2075628Z * [new tag] viable/strict/1763448826 -> viable/strict/1763448826 2025-12-04T09:23:40.2075761Z * [new tag] viable/strict/1763450717 -> viable/strict/1763450717 2025-12-04T09:23:40.2075904Z * [new tag] viable/strict/1763452183 -> viable/strict/1763452183 2025-12-04T09:23:40.2076036Z * [new tag] viable/strict/1763457945 -> viable/strict/1763457945 2025-12-04T09:23:40.2076178Z * [new tag] viable/strict/1763459439 -> viable/strict/1763459439 2025-12-04T09:23:40.2076321Z * [new tag] viable/strict/1763461556 -> viable/strict/1763461556 2025-12-04T09:23:40.2076461Z * [new tag] viable/strict/1763463103 -> viable/strict/1763463103 2025-12-04T09:23:40.2078859Z * [new tag] viable/strict/1763465100 -> viable/strict/1763465100 2025-12-04T09:23:40.2079453Z * [new tag] viable/strict/1763468866 -> viable/strict/1763468866 2025-12-04T09:23:40.2079635Z * [new tag] viable/strict/1763493823 -> viable/strict/1763493823 2025-12-04T09:23:40.2079771Z * [new tag] viable/strict/1763496249 -> viable/strict/1763496249 2025-12-04T09:23:40.2079902Z * [new tag] viable/strict/1763502620 -> viable/strict/1763502620 2025-12-04T09:23:40.2080038Z * [new tag] viable/strict/1763504715 -> viable/strict/1763504715 2025-12-04T09:23:40.2080187Z * [new tag] viable/strict/1763506208 -> viable/strict/1763506208 2025-12-04T09:23:40.2080604Z * [new tag] viable/strict/1763520590 -> viable/strict/1763520590 2025-12-04T09:23:40.2080766Z * [new tag] viable/strict/1763523357 -> viable/strict/1763523357 2025-12-04T09:23:40.2080911Z * [new tag] viable/strict/1763529922 -> viable/strict/1763529922 2025-12-04T09:23:40.2081044Z * [new tag] viable/strict/1763531408 -> viable/strict/1763531408 2025-12-04T09:23:40.2085951Z * [new tag] viable/strict/1763533622 -> viable/strict/1763533622 2025-12-04T09:23:40.2086271Z * [new tag] viable/strict/1763538576 -> viable/strict/1763538576 2025-12-04T09:23:40.2088555Z * [new tag] viable/strict/1763545823 -> viable/strict/1763545823 2025-12-04T09:23:40.2088881Z * [new tag] viable/strict/1763547951 -> viable/strict/1763547951 2025-12-04T09:23:40.2089084Z * [new tag] viable/strict/1763551477 -> viable/strict/1763551477 2025-12-04T09:23:40.2089229Z * [new tag] viable/strict/1763552982 -> viable/strict/1763552982 2025-12-04T09:23:40.2089453Z * [new tag] viable/strict/1763594698 -> viable/strict/1763594698 2025-12-04T09:23:40.2089609Z * [new tag] viable/strict/1763596178 -> viable/strict/1763596178 2025-12-04T09:23:40.2090316Z * [new tag] viable/strict/1763599155 -> viable/strict/1763599155 2025-12-04T09:23:40.2090491Z * [new tag] viable/strict/1763603717 -> viable/strict/1763603717 2025-12-04T09:23:40.2090723Z * [new tag] viable/strict/1763606923 -> viable/strict/1763606923 2025-12-04T09:23:40.2092548Z * [new tag] viable/strict/1763609715 -> viable/strict/1763609715 2025-12-04T09:23:40.2092719Z * [new tag] viable/strict/1763612757 -> viable/strict/1763612757 2025-12-04T09:23:40.2093137Z * [new tag] viable/strict/1763616325 -> viable/strict/1763616325 2025-12-04T09:23:40.2093276Z * [new tag] viable/strict/1763623509 -> viable/strict/1763623509 2025-12-04T09:23:40.2093492Z * [new tag] viable/strict/1763624984 -> viable/strict/1763624984 2025-12-04T09:23:40.2093681Z * [new tag] viable/strict/1763628796 -> viable/strict/1763628796 2025-12-04T09:23:40.2093816Z * [new tag] viable/strict/1763634343 -> viable/strict/1763634343 2025-12-04T09:23:40.2099617Z * [new tag] viable/strict/1763635867 -> viable/strict/1763635867 2025-12-04T09:23:40.2099790Z * [new tag] viable/strict/1763639382 -> viable/strict/1763639382 2025-12-04T09:23:40.2099930Z * [new tag] viable/strict/1763646626 -> viable/strict/1763646626 2025-12-04T09:23:40.2100056Z * [new tag] viable/strict/1763655997 -> viable/strict/1763655997 2025-12-04T09:23:40.2100215Z * [new tag] viable/strict/1763659444 -> viable/strict/1763659444 2025-12-04T09:23:40.2100349Z * [new tag] viable/strict/1763660992 -> viable/strict/1763660992 2025-12-04T09:23:40.2100477Z * [new tag] viable/strict/1763663201 -> viable/strict/1763663201 2025-12-04T09:23:40.2100630Z * [new tag] viable/strict/1763670362 -> viable/strict/1763670362 2025-12-04T09:23:40.2100755Z * [new tag] viable/strict/1763675378 -> viable/strict/1763675378 2025-12-04T09:23:40.2100945Z * [new tag] viable/strict/1763693343 -> viable/strict/1763693343 2025-12-04T09:23:40.2101099Z * [new tag] viable/strict/1763696088 -> viable/strict/1763696088 2025-12-04T09:23:40.2101225Z * [new tag] viable/strict/1763697343 -> viable/strict/1763697343 2025-12-04T09:23:40.2101500Z * [new tag] viable/strict/1763699165 -> viable/strict/1763699165 2025-12-04T09:23:40.2102002Z * [new tag] viable/strict/1763700660 -> viable/strict/1763700660 2025-12-04T09:23:40.2102159Z * [new tag] viable/strict/1763704209 -> viable/strict/1763704209 2025-12-04T09:23:40.2102303Z * [new tag] viable/strict/1763706411 -> viable/strict/1763706411 2025-12-04T09:23:40.2108448Z * [new tag] viable/strict/1763708082 -> viable/strict/1763708082 2025-12-04T09:23:40.2108782Z * [new tag] viable/strict/1763711381 -> viable/strict/1763711381 2025-12-04T09:23:40.2108979Z * [new tag] viable/strict/1763713593 -> viable/strict/1763713593 2025-12-04T09:23:40.2109155Z * [new tag] viable/strict/1763715201 -> viable/strict/1763715201 2025-12-04T09:23:40.2109337Z * [new tag] viable/strict/1763733017 -> viable/strict/1763733017 2025-12-04T09:23:40.2109500Z * [new tag] viable/strict/1763735108 -> viable/strict/1763735108 2025-12-04T09:23:40.2109798Z * [new tag] viable/strict/1763749579 -> viable/strict/1763749579 2025-12-04T09:23:40.2110419Z * [new tag] viable/strict/1763751113 -> viable/strict/1763751113 2025-12-04T09:23:40.2110580Z * [new tag] viable/strict/1763753035 -> viable/strict/1763753035 2025-12-04T09:23:40.2115076Z * [new tag] viable/strict/1763754578 -> viable/strict/1763754578 2025-12-04T09:23:40.2115785Z * [new tag] viable/strict/1763756748 -> viable/strict/1763756748 2025-12-04T09:23:40.2115965Z * [new tag] viable/strict/1763758205 -> viable/strict/1763758205 2025-12-04T09:23:40.2116098Z * [new tag] viable/strict/1763764050 -> viable/strict/1763764050 2025-12-04T09:23:40.2116225Z * [new tag] viable/strict/1763771887 -> viable/strict/1763771887 2025-12-04T09:23:40.2116359Z * [new tag] viable/strict/1763773920 -> viable/strict/1763773920 2025-12-04T09:23:40.2116636Z * [new tag] viable/strict/1763776501 -> viable/strict/1763776501 2025-12-04T09:23:40.2116773Z * [new tag] viable/strict/1763779437 -> viable/strict/1763779437 2025-12-04T09:23:40.2116899Z * [new tag] viable/strict/1763781038 -> viable/strict/1763781038 2025-12-04T09:23:40.2117080Z * [new tag] viable/strict/1763782245 -> viable/strict/1763782245 2025-12-04T09:23:40.2117212Z * [new tag] viable/strict/1763785568 -> viable/strict/1763785568 2025-12-04T09:23:40.2117335Z * [new tag] viable/strict/1763787006 -> viable/strict/1763787006 2025-12-04T09:23:40.2117466Z * [new tag] viable/strict/1763789103 -> viable/strict/1763789103 2025-12-04T09:23:40.2117593Z * [new tag] viable/strict/1763790578 -> viable/strict/1763790578 2025-12-04T09:23:40.2117716Z * [new tag] viable/strict/1763796275 -> viable/strict/1763796275 2025-12-04T09:23:40.2117859Z * [new tag] viable/strict/1763801465 -> viable/strict/1763801465 2025-12-04T09:23:40.2117982Z * [new tag] viable/strict/1763803522 -> viable/strict/1763803522 2025-12-04T09:23:40.2118362Z * [new tag] viable/strict/1763808581 -> viable/strict/1763808581 2025-12-04T09:23:40.2118539Z * [new tag] viable/strict/1763840977 -> viable/strict/1763840977 2025-12-04T09:23:40.2118775Z * [new tag] viable/strict/1763846659 -> viable/strict/1763846659 2025-12-04T09:23:40.2120406Z * [new tag] viable/strict/1763872065 -> viable/strict/1763872065 2025-12-04T09:23:40.2120722Z * [new tag] viable/strict/1763873648 -> viable/strict/1763873648 2025-12-04T09:23:40.2125272Z * [new tag] viable/strict/1763875506 -> viable/strict/1763875506 2025-12-04T09:23:40.2125566Z * [new tag] viable/strict/1763889904 -> viable/strict/1763889904 2025-12-04T09:23:40.2125838Z * [new tag] viable/strict/1763930999 -> viable/strict/1763930999 2025-12-04T09:23:40.2125977Z * [new tag] viable/strict/1763944964 -> viable/strict/1763944964 2025-12-04T09:23:40.2126186Z * [new tag] viable/strict/1763958474 -> viable/strict/1763958474 2025-12-04T09:23:40.2126333Z * [new tag] viable/strict/1763967263 -> viable/strict/1763967263 2025-12-04T09:23:40.2126541Z * [new tag] viable/strict/1763972803 -> viable/strict/1763972803 2025-12-04T09:23:40.2126722Z * [new tag] viable/strict/1763976376 -> viable/strict/1763976376 2025-12-04T09:23:40.2126930Z * [new tag] viable/strict/1763989404 -> viable/strict/1763989404 2025-12-04T09:23:40.2127075Z * [new tag] viable/strict/1763990887 -> viable/strict/1763990887 2025-12-04T09:23:40.2127276Z * [new tag] viable/strict/1764019919 -> viable/strict/1764019919 2025-12-04T09:23:40.2127799Z * [new tag] viable/strict/1764023134 -> viable/strict/1764023134 2025-12-04T09:23:40.2128223Z * [new tag] viable/strict/1764024593 -> viable/strict/1764024593 2025-12-04T09:23:40.2131246Z * [new tag] viable/strict/1764026706 -> viable/strict/1764026706 2025-12-04T09:23:40.2131469Z * [new tag] viable/strict/1764031139 -> viable/strict/1764031139 2025-12-04T09:23:40.2131611Z * [new tag] viable/strict/1764033131 -> viable/strict/1764033131 2025-12-04T09:23:40.2131741Z * [new tag] viable/strict/1764035725 -> viable/strict/1764035725 2025-12-04T09:23:40.2131880Z * [new tag] viable/strict/1764624265 -> viable/strict/1764624265 2025-12-04T09:23:40.2132039Z * [new tag] viable/strict/1764631514 -> viable/strict/1764631514 2025-12-04T09:23:40.2132656Z * [new tag] viable/strict/1764632987 -> viable/strict/1764632987 2025-12-04T09:23:40.2133246Z * [new tag] viable/strict/1764636063 -> viable/strict/1764636063 2025-12-04T09:23:40.2133655Z * [new tag] viable/strict/1764643975 -> viable/strict/1764643975 2025-12-04T09:23:40.2134375Z * [new tag] viable/strict/1764646859 -> viable/strict/1764646859 2025-12-04T09:23:40.2134877Z * [new tag] viable/strict/1764653120 -> viable/strict/1764653120 2025-12-04T09:23:40.2135349Z * [new tag] viable/strict/1764654632 -> viable/strict/1764654632 2025-12-04T09:23:40.2135869Z * [new tag] viable/strict/1764656821 -> viable/strict/1764656821 2025-12-04T09:23:40.2136396Z * [new tag] viable/strict/1764658557 -> viable/strict/1764658557 2025-12-04T09:23:40.2136882Z * [new tag] viable/strict/1764660333 -> viable/strict/1764660333 2025-12-04T09:23:40.2137510Z * [new tag] viable/strict/1764661812 -> viable/strict/1764661812 2025-12-04T09:23:40.2138916Z * [new tag] viable/strict/1764664023 -> viable/strict/1764664023 2025-12-04T09:23:40.2139054Z * [new tag] viable/strict/1764669150 -> viable/strict/1764669150 2025-12-04T09:23:40.2139187Z * [new tag] viable/strict/1764680709 -> viable/strict/1764680709 2025-12-04T09:23:40.2140051Z * [new tag] viable/strict/1764687619 -> viable/strict/1764687619 2025-12-04T09:23:40.2140205Z * [new tag] viable/strict/1764696355 -> viable/strict/1764696355 2025-12-04T09:23:40.2144145Z * [new tag] viable/strict/1764701767 -> viable/strict/1764701767 2025-12-04T09:23:40.2144321Z * [new tag] viable/strict/1764710768 -> viable/strict/1764710768 2025-12-04T09:23:40.2144470Z * [new tag] viable/strict/1764716202 -> viable/strict/1764716202 2025-12-04T09:23:40.2144624Z * [new tag] viable/strict/1764793566 -> viable/strict/1764793566 2025-12-04T09:23:40.2144763Z * [new tag] viable/strict/1764797093 -> viable/strict/1764797093 2025-12-04T09:23:40.2144894Z * [new tag] viable/strict/1764800729 -> viable/strict/1764800729 2025-12-04T09:23:40.2145013Z * [new tag] whc_flight_1 -> whc_flight_1 2025-12-04T09:23:40.2145145Z * [new tag] whc_flight_2 -> whc_flight_2 2025-12-04T09:23:40.2150176Z * [new tag] whc_flight_4 -> whc_flight_4 2025-12-04T09:23:40.2665734Z [command]/usr/bin/git rev-parse --verify --quiet ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32^{object} 2025-12-04T09:23:40.2696261Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:23:40.2711991Z ##[endgroup] 2025-12-04T09:23:40.2713918Z ##[group]Determining the checkout info 2025-12-04T09:23:40.2714350Z ##[endgroup] 2025-12-04T09:23:40.2714584Z [command]/usr/bin/git sparse-checkout disable 2025-12-04T09:23:40.2742966Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-12-04T09:23:40.2791236Z ##[group]Checking out the ref 2025-12-04T09:23:40.2795991Z [command]/usr/bin/git checkout --progress --force ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:23:41.3105987Z Updating files: 95% (19117/20121) 2025-12-04T09:23:41.3221909Z Updating files: 96% (19317/20121) 2025-12-04T09:23:41.3349876Z Updating files: 97% (19518/20121) 2025-12-04T09:23:41.3578511Z Updating files: 98% (19719/20121) 2025-12-04T09:23:41.3714216Z Updating files: 99% (19920/20121) 2025-12-04T09:23:41.3715789Z Updating files: 100% (20121/20121) 2025-12-04T09:23:41.3716044Z Updating files: 100% (20121/20121), done. 2025-12-04T09:23:41.3938218Z Note: switching to 'ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32'. 2025-12-04T09:23:41.3943410Z 2025-12-04T09:23:41.3945628Z You are in 'detached HEAD' state. You can look around, make experimental 2025-12-04T09:23:41.3946063Z changes and commit them, and you can discard any commits you make in this 2025-12-04T09:23:41.3946785Z state without impacting any branches by switching back to a branch. 2025-12-04T09:23:41.3947030Z 2025-12-04T09:23:41.3947369Z If you want to create a new branch to retain commits you create, you may 2025-12-04T09:23:41.3947741Z do so (now or later) by using -c with the switch command. Example: 2025-12-04T09:23:41.3947966Z 2025-12-04T09:23:41.3948209Z git switch -c 2025-12-04T09:23:41.3948359Z 2025-12-04T09:23:41.3948444Z Or undo this operation with: 2025-12-04T09:23:41.3948578Z 2025-12-04T09:23:41.3948646Z git switch - 2025-12-04T09:23:41.3948744Z 2025-12-04T09:23:41.3948908Z Turn off this advice by setting config variable advice.detachedHead to false 2025-12-04T09:23:41.3949122Z 2025-12-04T09:23:41.3949320Z HEAD is now at ffd9b0fb435 Resolve collective autotuning test failure on arm (#168919) 2025-12-04T09:23:41.3994887Z ##[endgroup] 2025-12-04T09:23:41.3995313Z ##[group]Setting up auth for fetching submodules 2025-12-04T09:23:41.4001021Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-12-04T09:23:41.4075219Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-12-04T09:23:41.4112035Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-12-04T09:23:41.4135585Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-12-04T09:23:41.4160025Z ##[endgroup] 2025-12-04T09:23:41.4160394Z ##[group]Fetching submodules 2025-12-04T09:23:41.4160664Z [command]/usr/bin/git submodule sync --recursive 2025-12-04T09:23:41.4559019Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-12-04T09:23:41.4911076Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-12-04T09:23:41.4916039Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-12-04T09:23:41.4918480Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-12-04T09:23:41.4919128Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-12-04T09:23:41.4919706Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-12-04T09:23:41.4920714Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-12-04T09:23:41.4921484Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-12-04T09:23:41.4922031Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-12-04T09:23:41.4922605Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-12-04T09:23:41.4923259Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-12-04T09:23:41.4928666Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-12-04T09:23:41.4929339Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-12-04T09:23:41.4929978Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-12-04T09:23:41.4930616Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-12-04T09:23:41.4931176Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-12-04T09:23:41.4936007Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-12-04T09:23:41.4936657Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-12-04T09:23:41.4937681Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-12-04T09:23:41.4942122Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:23:41.4942727Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-12-04T09:23:41.4945083Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-12-04T09:23:41.4946568Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-12-04T09:23:41.4954655Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-12-04T09:23:41.4955374Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-12-04T09:23:41.4955956Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-12-04T09:23:41.4956549Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-12-04T09:23:41.4958875Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-12-04T09:23:41.4963919Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-12-04T09:23:41.4967445Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-12-04T09:23:41.4968446Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-12-04T09:23:41.4974778Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-12-04T09:23:41.4980363Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-12-04T09:23:41.4981246Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-12-04T09:23:41.4981987Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-12-04T09:23:41.4982569Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-12-04T09:23:41.4987198Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-12-04T09:23:41.4993482Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-12-04T09:23:41.5017576Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-12-04T09:23:41.7318445Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-12-04T09:23:41.7318954Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-12-04T09:23:41.7319409Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-12-04T09:23:41.7319860Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-12-04T09:23:41.7326456Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-12-04T09:23:42.0163200Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-12-04T09:23:42.0164321Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-12-04T09:23:42.0165140Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-12-04T09:23:42.0165928Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-12-04T09:23:42.0166860Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-12-04T09:23:42.0167640Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-12-04T09:23:42.0168391Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-12-04T09:23:42.0356618Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-12-04T09:23:42.5451773Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-12-04T09:23:42.5453147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-12-04T09:23:42.5454530Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-12-04T09:23:42.5455389Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-12-04T09:23:42.5456237Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-12-04T09:23:42.5457074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-12-04T09:23:42.6453071Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-12-04T09:23:43.3625383Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-12-04T09:23:43.3626297Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-12-04T09:23:43.3627115Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-12-04T09:23:43.3627959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-12-04T09:23:43.3628772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-12-04T09:23:43.3629575Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-12-04T09:23:43.4397947Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-12-04T09:23:57.9805791Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-12-04T09:23:57.9806610Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-12-04T09:23:57.9807083Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-12-04T09:23:57.9807534Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-12-04T09:23:57.9808020Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-12-04T09:23:57.9808455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-12-04T09:23:57.9808883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-12-04T09:23:57.9809326Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-12-04T09:23:58.0806872Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-12-04T09:23:58.1973557Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-12-04T09:23:58.2086410Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-12-04T09:23:58.2179135Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-12-04T09:23:58.2388610Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-12-04T09:23:58.3109099Z Submodule path 'third_party/NVTX': checked out '3ebbc93ded7285963bff932c678fa367eb393ba6' 2025-12-04T09:23:58.3558627Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-12-04T09:23:58.8850124Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-12-04T09:23:59.0406445Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-12-04T09:23:59.0423805Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:23:59.0446179Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-12-04T09:24:04.2228626Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-12-04T09:24:04.2441217Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-12-04T09:24:04.5244342Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-12-04T09:24:04.5679478Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246' 2025-12-04T09:24:04.6555274Z Submodule path 'third_party/cpuinfo': checked out 'f858c30bcb16f8effd5ff46996f0514539e17abc' 2025-12-04T09:24:04.6994577Z Submodule path 'third_party/cudnn_frontend': checked out '0b1577c8c83401237d601d0d0db5210506705396' 2025-12-04T09:24:05.2340135Z Submodule path 'third_party/cutlass': checked out 'f88806b1e31dfa579842638740216dd41fc6c588' 2025-12-04T09:24:05.3572302Z Submodule path 'third_party/fbgemm': checked out 'c0b988d39a9e47c794d699f29930ed4d7c7e13a4' 2025-12-04T09:24:05.3588472Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-12-04T09:24:05.3589451Z Submodule 'external/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:24:05.3592040Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:24:05.3592773Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-12-04T09:24:05.3597971Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-12-04T09:24:05.3598889Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:24:05.3599638Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-12-04T09:24:05.3626813Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-12-04T09:24:06.5877127Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-12-04T09:24:06.5877947Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-12-04T09:24:06.5878680Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-12-04T09:24:06.6879036Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-12-04T09:24:10.2188634Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-12-04T09:24:10.3193605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-12-04T09:24:12.3463707Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-12-04T09:24:12.6440610Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-12-04T09:24:12.7411030Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-12-04T09:24:13.3082265Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '98125ce499b0fdf7ffbe0e3052f5b8709f4840f8' 2025-12-04T09:24:13.3516936Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:24:13.3628621Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691' 2025-12-04T09:24:13.4570529Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-12-04T09:24:13.5205719Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-12-04T09:24:13.5222649Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:24:13.5223635Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:24:13.5258948Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-12-04T09:24:18.4496304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-12-04T09:24:18.6419199Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-12-04T09:24:19.1323984Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-12-04T09:24:19.2476698Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-12-04T09:24:19.2748104Z Submodule path 'third_party/fmt': checked out '407c905e45ad75fc29bf0f9bb7c5c2fd3475976f' 2025-12-04T09:24:19.3089463Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-12-04T09:24:19.3302696Z Submodule path 'third_party/gloo': checked out '54cbae0d3a67fa890b4c3d9ee162b7860315e341' 2025-12-04T09:24:19.3686390Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:24:19.3807892Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-12-04T09:24:19.3823802Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-12-04T09:24:19.3846939Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-12-04T09:24:35.9937897Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-12-04T09:24:36.0111958Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-12-04T09:24:36.0986915Z Submodule path 'third_party/kineto': checked out '31f85df8fbd89c188f14ef10f1ec65379786b943' 2025-12-04T09:24:36.0999004Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:24:36.1000025Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:24:36.1001608Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:24:36.1029757Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-12-04T09:24:37.3889124Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-12-04T09:24:37.5785717Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-12-04T09:24:37.6546965Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out 'd2ffe0a4e3acace628db49974246b66fc3e85fb1' 2025-12-04T09:24:37.6567260Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:24:37.6571951Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:24:37.6572805Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:24:37.6592360Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:24:37.6595022Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:24:37.6595950Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:24:37.6596808Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:24:37.6597748Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:24:37.6601033Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:24:37.6607690Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-12-04T09:24:39.4478917Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-12-04T09:24:39.4479753Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-12-04T09:24:39.4480793Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'... 2025-12-04T09:24:39.4481525Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-12-04T09:24:39.4482184Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-12-04T09:24:39.4482854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-12-04T09:24:39.4483482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-12-04T09:24:39.5481429Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-12-04T09:24:44.5973322Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-12-04T09:24:44.6125910Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-12-04T09:24:44.6438006Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-12-04T09:24:44.6560435Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-12-04T09:24:44.6573594Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:24:44.6603691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-12-04T09:24:45.2607175Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-12-04T09:24:45.2774063Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-12-04T09:24:45.3170570Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:24:45.3994644Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-12-04T09:24:45.4148655Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-12-04T09:24:45.4298291Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp': checked out 'b1234816facfdda29845c46696a02998a4af115a' 2025-12-04T09:24:45.4310912Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:24:45.4311862Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:24:45.4340231Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-12-04T09:24:47.2728568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'... 2025-12-04T09:24:47.4838477Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'd7ba35bbb649209c66e582d5a0244ba988a15159' 2025-12-04T09:24:47.5229466Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-12-04T09:24:47.5511411Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-12-04T09:24:47.5894532Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:24:47.6344966Z Submodule path 'third_party/kleidiai': checked out 'd7770c89632329a9914ef1a90289917597639cbe' 2025-12-04T09:24:47.6678702Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-12-04T09:24:47.7561701Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-12-04T09:24:48.0368888Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-12-04T09:24:48.0397046Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-12-04T09:24:48.0424862Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-12-04T09:24:48.8828038Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-12-04T09:24:48.9363284Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-12-04T09:24:48.9381812Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:24:48.9382612Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:24:48.9383640Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:24:48.9384357Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:24:48.9385187Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:24:48.9390254Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:24:48.9391123Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:24:48.9391822Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:24:48.9421469Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-12-04T09:24:49.3199894Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-12-04T09:24:49.3200997Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-12-04T09:24:49.3201695Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-12-04T09:24:49.3202403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-12-04T09:24:49.4198803Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-12-04T09:24:49.9450844Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-12-04T09:24:55.7066037Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-12-04T09:24:56.5333403Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-12-04T09:24:56.5679795Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-12-04T09:24:56.5835336Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-12-04T09:24:56.6720124Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-12-04T09:24:56.6851267Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-12-04T09:24:56.6982384Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-12-04T09:24:56.7121849Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-12-04T09:24:56.7137381Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:24:56.7138202Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:24:56.7167944Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-12-04T09:24:58.4946504Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-12-04T09:24:58.7053481Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-12-04T09:24:58.7455055Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-12-04T09:24:59.0633714Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-12-04T09:24:59.0752162Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-12-04T09:24:59.2930814Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-12-04T09:24:59.2946872Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:24:59.2947980Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-12-04T09:24:59.2972227Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-12-04T09:24:59.7811126Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-12-04T09:25:00.2033397Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-12-04T09:25:00.2672750Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-12-04T09:25:00.2767320Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-12-04T09:25:00.2884863Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-12-04T09:25:00.3260480Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8' 2025-12-04T09:25:00.3512041Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-12-04T09:25:00.3928491Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-12-04T09:25:00.4161598Z Submodule path 'third_party/tensorpipe': checked out '2b4cd91092d335a697416b2a3cb398283246849d' 2025-12-04T09:25:00.4177655Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:25:00.4178432Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:25:00.4184622Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:25:00.4185350Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:25:00.4212733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-12-04T09:25:01.2977382Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-12-04T09:25:01.2978043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-12-04T09:25:01.3980000Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-12-04T09:25:01.4557777Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-12-04T09:25:01.4696047Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-12-04T09:25:01.5340030Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-12-04T09:25:01.5599316Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-12-04T09:25:01.5614200Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:25:01.5639011Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-12-04T09:25:01.7319220Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-12-04T09:25:01.7361884Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-12-04T09:25:01.7663326Z Entering 'android/libs/fbjni' 2025-12-04T09:25:01.7702284Z Entering 'third_party/FP16' 2025-12-04T09:25:01.7740988Z Entering 'third_party/FXdiv' 2025-12-04T09:25:01.7795404Z Entering 'third_party/NNPACK' 2025-12-04T09:25:01.7831299Z Entering 'third_party/NVTX' 2025-12-04T09:25:01.7873338Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:25:01.7919219Z Entering 'third_party/XNNPACK' 2025-12-04T09:25:01.7972863Z Entering 'third_party/aiter' 2025-12-04T09:25:01.8011815Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:25:01.8061993Z Entering 'third_party/benchmark' 2025-12-04T09:25:01.8109143Z Entering 'third_party/composable_kernel' 2025-12-04T09:25:01.8152086Z Entering 'third_party/cpp-httplib' 2025-12-04T09:25:01.8194814Z Entering 'third_party/cpuinfo' 2025-12-04T09:25:01.8236303Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:25:01.8273352Z Entering 'third_party/cutlass' 2025-12-04T09:25:01.8319030Z Entering 'third_party/fbgemm' 2025-12-04T09:25:01.8365798Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:25:01.8411569Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:25:01.8459745Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:25:01.8497782Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:25:01.8541495Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:25:01.8582945Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:25:01.8623613Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:25:01.8669198Z Entering 'third_party/flash-attention' 2025-12-04T09:25:01.8704326Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:25:01.8743682Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:25:01.8791928Z Entering 'third_party/flatbuffers' 2025-12-04T09:25:01.8836818Z Entering 'third_party/fmt' 2025-12-04T09:25:01.8879819Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:25:01.8921192Z Entering 'third_party/gloo' 2025-12-04T09:25:01.8962890Z Entering 'third_party/googletest' 2025-12-04T09:25:01.9008346Z Entering 'third_party/ideep' 2025-12-04T09:25:01.9050284Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:25:01.9102775Z Entering 'third_party/ittapi' 2025-12-04T09:25:01.9140640Z Entering 'third_party/kineto' 2025-12-04T09:25:01.9180417Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:25:01.9215759Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:25:01.9262414Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:25:01.9297046Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:25:01.9336992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:25:01.9377144Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:25:01.9418655Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:25:01.9457799Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:25:01.9495179Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:25:01.9535966Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:25:01.9575280Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:25:01.9612339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:01.9654227Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:01.9702746Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:25:01.9744627Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:25:01.9788995Z Entering 'third_party/kleidiai' 2025-12-04T09:25:01.9830063Z Entering 'third_party/mimalloc' 2025-12-04T09:25:01.9876382Z Entering 'third_party/nlohmann' 2025-12-04T09:25:01.9913948Z Entering 'third_party/onnx' 2025-12-04T09:25:01.9971204Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:25:02.0011375Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:25:02.0051155Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:25:02.0090719Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:25:02.0131991Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:25:02.0171790Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:25:02.0212172Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:25:02.0249667Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:25:02.0292618Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:25:02.0327199Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:02.0368222Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:02.0412720Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:25:02.0469620Z Entering 'third_party/pocketfft' 2025-12-04T09:25:02.0506069Z Entering 'third_party/protobuf' 2025-12-04T09:25:02.0550211Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:25:02.0586539Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:25:02.0632377Z Entering 'third_party/psimd' 2025-12-04T09:25:02.0676720Z Entering 'third_party/pthreadpool' 2025-12-04T09:25:02.0718532Z Entering 'third_party/pybind11' 2025-12-04T09:25:02.0753994Z Entering 'third_party/python-peachpy' 2025-12-04T09:25:02.0792758Z Entering 'third_party/sleef' 2025-12-04T09:25:02.0833136Z Entering 'third_party/tensorpipe' 2025-12-04T09:25:02.0874811Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:25:02.0915533Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:25:02.0957131Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:25:02.0992429Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:25:02.1034539Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:25:02.1101350Z ##[endgroup] 2025-12-04T09:25:02.1106717Z ##[group]Persisting credentials for submodules 2025-12-04T09:25:02.1110031Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-12-04T09:25:02.1397299Z Entering 'android/libs/fbjni' 2025-12-04T09:25:02.1448436Z Entering 'third_party/FP16' 2025-12-04T09:25:02.1507791Z Entering 'third_party/FXdiv' 2025-12-04T09:25:02.1558536Z Entering 'third_party/NNPACK' 2025-12-04T09:25:02.1615100Z Entering 'third_party/NVTX' 2025-12-04T09:25:02.1672876Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:25:02.1731214Z Entering 'third_party/XNNPACK' 2025-12-04T09:25:02.1798895Z Entering 'third_party/aiter' 2025-12-04T09:25:02.1858220Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:25:02.1919036Z Entering 'third_party/benchmark' 2025-12-04T09:25:02.1968188Z Entering 'third_party/composable_kernel' 2025-12-04T09:25:02.2034160Z Entering 'third_party/cpp-httplib' 2025-12-04T09:25:02.2081181Z Entering 'third_party/cpuinfo' 2025-12-04T09:25:02.2129904Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:25:02.2188646Z Entering 'third_party/cutlass' 2025-12-04T09:25:02.2245076Z Entering 'third_party/fbgemm' 2025-12-04T09:25:02.2298004Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:25:02.2350581Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:25:02.2411978Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:25:02.2463781Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:25:02.2524608Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:25:02.2578904Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:25:02.2629685Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:25:02.2686182Z Entering 'third_party/flash-attention' 2025-12-04T09:25:02.2738912Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:25:02.2801666Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:25:02.2870240Z Entering 'third_party/flatbuffers' 2025-12-04T09:25:02.2919939Z Entering 'third_party/fmt' 2025-12-04T09:25:02.2968177Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:25:02.3023562Z Entering 'third_party/gloo' 2025-12-04T09:25:02.3077344Z Entering 'third_party/googletest' 2025-12-04T09:25:02.3133719Z Entering 'third_party/ideep' 2025-12-04T09:25:02.3184506Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:25:02.3242120Z Entering 'third_party/ittapi' 2025-12-04T09:25:02.3300889Z Entering 'third_party/kineto' 2025-12-04T09:25:02.3346243Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:25:02.3401671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:25:02.3455638Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:25:02.3512852Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:25:02.3571289Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:25:02.3624271Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:25:02.3689816Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:25:02.3735869Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:25:02.3796220Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:25:02.3852156Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:25:02.3907782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:25:02.3953735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:02.4011105Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:02.4072054Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:25:02.4120068Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:25:02.4176673Z Entering 'third_party/kleidiai' 2025-12-04T09:25:02.4233690Z Entering 'third_party/mimalloc' 2025-12-04T09:25:02.4289755Z Entering 'third_party/nlohmann' 2025-12-04T09:25:02.4345202Z Entering 'third_party/onnx' 2025-12-04T09:25:02.4409762Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:25:02.4468752Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:25:02.4521346Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:25:02.4581374Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:25:02.4632256Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:25:02.4693685Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:25:02.4748181Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:25:02.4797899Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:25:02.4856554Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:25:02.4910491Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:02.4963117Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:02.5016751Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:25:02.5091507Z Entering 'third_party/pocketfft' 2025-12-04T09:25:02.5142869Z Entering 'third_party/protobuf' 2025-12-04T09:25:02.5202669Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:25:02.5253802Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:25:02.5309466Z Entering 'third_party/psimd' 2025-12-04T09:25:02.5362001Z Entering 'third_party/pthreadpool' 2025-12-04T09:25:02.5418463Z Entering 'third_party/pybind11' 2025-12-04T09:25:02.5477990Z Entering 'third_party/python-peachpy' 2025-12-04T09:25:02.5529894Z Entering 'third_party/sleef' 2025-12-04T09:25:02.5579506Z Entering 'third_party/tensorpipe' 2025-12-04T09:25:02.5629924Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:25:02.5684276Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:25:02.5740875Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:25:02.5795974Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:25:02.5844038Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:25:02.5919923Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-12-04T09:25:02.6229725Z Entering 'android/libs/fbjni' 2025-12-04T09:25:02.6281382Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-12-04T09:25:02.6301573Z Entering 'third_party/FP16' 2025-12-04T09:25:02.6347525Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-12-04T09:25:02.6368280Z Entering 'third_party/FXdiv' 2025-12-04T09:25:02.6415871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-12-04T09:25:02.6430951Z Entering 'third_party/NNPACK' 2025-12-04T09:25:02.6478995Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-12-04T09:25:02.6500691Z Entering 'third_party/NVTX' 2025-12-04T09:25:02.6544178Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-12-04T09:25:02.6561674Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:25:02.6611773Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-12-04T09:25:02.6628798Z Entering 'third_party/XNNPACK' 2025-12-04T09:25:02.6679821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-12-04T09:25:02.6709106Z Entering 'third_party/aiter' 2025-12-04T09:25:02.6754409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-12-04T09:25:02.6773947Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:25:02.6821812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-12-04T09:25:02.6844657Z Entering 'third_party/benchmark' 2025-12-04T09:25:02.6899396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:25:02.6918006Z Entering 'third_party/composable_kernel' 2025-12-04T09:25:02.6969268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-12-04T09:25:02.6993594Z Entering 'third_party/cpp-httplib' 2025-12-04T09:25:02.7041745Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-12-04T09:25:02.7057651Z Entering 'third_party/cpuinfo' 2025-12-04T09:25:02.7110015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-12-04T09:25:02.7125189Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:25:02.7180782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-12-04T09:25:02.7197860Z Entering 'third_party/cutlass' 2025-12-04T09:25:02.7246445Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-12-04T09:25:02.7272146Z Entering 'third_party/fbgemm' 2025-12-04T09:25:02.7321187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-12-04T09:25:02.7340016Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:25:02.7388270Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-12-04T09:25:02.7408723Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:25:02.7453840Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-12-04T09:25:02.7473033Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:25:02.7522190Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-12-04T09:25:02.7537661Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:25:02.7584613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-12-04T09:25:02.7608126Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:25:02.7655937Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-12-04T09:25:02.7672549Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:25:02.7717847Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-12-04T09:25:02.7734458Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:25:02.7780825Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-12-04T09:25:02.7804598Z Entering 'third_party/flash-attention' 2025-12-04T09:25:02.7853848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-12-04T09:25:02.7865098Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:25:02.7918366Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-12-04T09:25:02.7935410Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:25:02.7987594Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-12-04T09:25:02.8010822Z Entering 'third_party/flatbuffers' 2025-12-04T09:25:02.8059291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-12-04T09:25:02.8078313Z Entering 'third_party/fmt' 2025-12-04T09:25:02.8122437Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:25:02.8138027Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:25:02.8187201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-12-04T09:25:02.8202458Z Entering 'third_party/gloo' 2025-12-04T09:25:02.8252478Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-12-04T09:25:02.8269813Z Entering 'third_party/googletest' 2025-12-04T09:25:02.8319984Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:25:02.8338314Z Entering 'third_party/ideep' 2025-12-04T09:25:02.8396053Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-12-04T09:25:02.8410912Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:25:02.8459188Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-12-04T09:25:02.8481516Z Entering 'third_party/ittapi' 2025-12-04T09:25:02.8530832Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-12-04T09:25:02.8547505Z Entering 'third_party/kineto' 2025-12-04T09:25:02.8593986Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-12-04T09:25:02.8611263Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:25:02.8662279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-12-04T09:25:02.8676892Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:25:02.8723760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-12-04T09:25:02.8740002Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:25:02.8791374Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-12-04T09:25:02.8814507Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:25:02.8859966Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:25:02.8874050Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:25:02.8926028Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-12-04T09:25:02.8944393Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:25:02.8991594Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-12-04T09:25:02.9013269Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:25:02.9057061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-12-04T09:25:02.9075255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:25:02.9128083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:25:02.9140638Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:25:02.9194541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-12-04T09:25:02.9211876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:25:02.9259543Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-12-04T09:25:02.9278607Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:25:02.9331219Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:25:02.9347468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:02.9397965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:25:02.9420339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:02.9466057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:25:02.9496302Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:25:02.9543757Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-12-04T09:25:02.9565264Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:25:02.9614371Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-12-04T09:25:02.9633095Z Entering 'third_party/kleidiai' 2025-12-04T09:25:02.9684018Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-12-04T09:25:02.9698007Z Entering 'third_party/mimalloc' 2025-12-04T09:25:02.9745850Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-12-04T09:25:02.9765926Z Entering 'third_party/nlohmann' 2025-12-04T09:25:02.9813778Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-12-04T09:25:02.9829636Z Entering 'third_party/onnx' 2025-12-04T09:25:02.9881065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-12-04T09:25:02.9909964Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:25:02.9958685Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:25:02.9977552Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:25:03.0026543Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-12-04T09:25:03.0047498Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:25:03.0098709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:25:03.0111105Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:25:03.0159299Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:25:03.0181102Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:25:03.0226367Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-12-04T09:25:03.0241043Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:25:03.0289836Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-12-04T09:25:03.0307941Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:25:03.0356043Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-12-04T09:25:03.0379185Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:25:03.0426666Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-12-04T09:25:03.0440642Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:25:03.0491287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:25:03.0507919Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:03.0552960Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:25:03.0572152Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:03.0621588Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:25:03.0638378Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:25:03.0686687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-12-04T09:25:03.0716291Z Entering 'third_party/pocketfft' 2025-12-04T09:25:03.0768742Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-12-04T09:25:03.0780557Z Entering 'third_party/protobuf' 2025-12-04T09:25:03.0827516Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-12-04T09:25:03.0844319Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:25:03.0893015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:25:03.0910083Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:25:03.0959856Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:25:03.0979080Z Entering 'third_party/psimd' 2025-12-04T09:25:03.1027840Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-12-04T09:25:03.1042256Z Entering 'third_party/pthreadpool' 2025-12-04T09:25:03.1092686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-12-04T09:25:03.1111531Z Entering 'third_party/pybind11' 2025-12-04T09:25:03.1158914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:25:03.1180345Z Entering 'third_party/python-peachpy' 2025-12-04T09:25:03.1223208Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-12-04T09:25:03.1238319Z Entering 'third_party/sleef' 2025-12-04T09:25:03.1289908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-12-04T09:25:03.1315214Z Entering 'third_party/tensorpipe' 2025-12-04T09:25:03.1358046Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-12-04T09:25:03.1376877Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:25:03.1424709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:25:03.1437091Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:25:03.1486333Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-12-04T09:25:03.1507263Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:25:03.1556411Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-12-04T09:25:03.1574683Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:25:03.1622381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:25:03.1634289Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:25:03.1679314Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-12-04T09:25:03.3347465Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-12-04T09:25:03.3677226Z Entering 'android/libs/fbjni' 2025-12-04T09:25:03.3717716Z Entering 'third_party/FP16' 2025-12-04T09:25:03.3761934Z Entering 'third_party/FXdiv' 2025-12-04T09:25:03.3798152Z Entering 'third_party/NNPACK' 2025-12-04T09:25:03.3838723Z Entering 'third_party/NVTX' 2025-12-04T09:25:03.3879029Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:25:03.3920093Z Entering 'third_party/XNNPACK' 2025-12-04T09:25:03.3970371Z Entering 'third_party/aiter' 2025-12-04T09:25:03.4010461Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:25:03.4054990Z Entering 'third_party/benchmark' 2025-12-04T09:25:03.4097966Z Entering 'third_party/composable_kernel' 2025-12-04T09:25:03.4143982Z Entering 'third_party/cpp-httplib' 2025-12-04T09:25:03.4187820Z Entering 'third_party/cpuinfo' 2025-12-04T09:25:03.4227779Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:25:03.4270389Z Entering 'third_party/cutlass' 2025-12-04T09:25:03.4320767Z Entering 'third_party/fbgemm' 2025-12-04T09:25:03.4359770Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:25:03.4402485Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:25:03.4445527Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:25:03.4484198Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:25:03.4530872Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:25:03.4569655Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:25:03.4602760Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:25:03.4645831Z Entering 'third_party/flash-attention' 2025-12-04T09:25:03.4689418Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:25:03.4736243Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:25:03.4783353Z Entering 'third_party/flatbuffers' 2025-12-04T09:25:03.4830195Z Entering 'third_party/fmt' 2025-12-04T09:25:03.4867147Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:25:03.4907059Z Entering 'third_party/gloo' 2025-12-04T09:25:03.4946677Z Entering 'third_party/googletest' 2025-12-04T09:25:03.4984793Z Entering 'third_party/ideep' 2025-12-04T09:25:03.5023586Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:25:03.5072675Z Entering 'third_party/ittapi' 2025-12-04T09:25:03.5109469Z Entering 'third_party/kineto' 2025-12-04T09:25:03.5143966Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:25:03.5185239Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:25:03.5226777Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:25:03.5269991Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:25:03.5310350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:25:03.5345565Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:25:03.5388478Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:25:03.5429856Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:25:03.5473828Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:25:03.5514268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:25:03.5555260Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:25:03.5594554Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:03.5632987Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:03.5681788Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:25:03.5728768Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:25:03.5771060Z Entering 'third_party/kleidiai' 2025-12-04T09:25:03.5813693Z Entering 'third_party/mimalloc' 2025-12-04T09:25:03.5849541Z Entering 'third_party/nlohmann' 2025-12-04T09:25:03.5892152Z Entering 'third_party/onnx' 2025-12-04T09:25:03.5944366Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:25:03.5995016Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:25:03.6035916Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:25:03.6079509Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:25:03.6115978Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:25:03.6157781Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:25:03.6202625Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:25:03.6244393Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:25:03.6290244Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:25:03.6331189Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:03.6368257Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:03.6412613Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:25:03.6474701Z Entering 'third_party/pocketfft' 2025-12-04T09:25:03.6516383Z Entering 'third_party/protobuf' 2025-12-04T09:25:03.6556378Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:25:03.6593615Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:25:03.6635675Z Entering 'third_party/psimd' 2025-12-04T09:25:03.6682459Z Entering 'third_party/pthreadpool' 2025-12-04T09:25:03.6722932Z Entering 'third_party/pybind11' 2025-12-04T09:25:03.6758992Z Entering 'third_party/python-peachpy' 2025-12-04T09:25:03.6803764Z Entering 'third_party/sleef' 2025-12-04T09:25:03.6838169Z Entering 'third_party/tensorpipe' 2025-12-04T09:25:03.6880541Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:25:03.6922279Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:25:03.6961273Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:25:03.6999522Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:25:03.7036764Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:25:03.7101960Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-12-04T09:25:03.7417745Z Entering 'android/libs/fbjni' 2025-12-04T09:25:03.7461730Z Entering 'third_party/FP16' 2025-12-04T09:25:03.7505992Z Entering 'third_party/FXdiv' 2025-12-04T09:25:03.7541639Z Entering 'third_party/NNPACK' 2025-12-04T09:25:03.7585298Z Entering 'third_party/NVTX' 2025-12-04T09:25:03.7627714Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:25:03.7664975Z Entering 'third_party/XNNPACK' 2025-12-04T09:25:03.7717212Z Entering 'third_party/aiter' 2025-12-04T09:25:03.7765599Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:25:03.7812890Z Entering 'third_party/benchmark' 2025-12-04T09:25:03.7853646Z Entering 'third_party/composable_kernel' 2025-12-04T09:25:03.7898649Z Entering 'third_party/cpp-httplib' 2025-12-04T09:25:03.7934758Z Entering 'third_party/cpuinfo' 2025-12-04T09:25:03.7978824Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:25:03.8016117Z Entering 'third_party/cutlass' 2025-12-04T09:25:03.8065656Z Entering 'third_party/fbgemm' 2025-12-04T09:25:03.8102784Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:25:03.8143738Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:25:03.8193475Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:25:03.8225380Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:25:03.8269900Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:25:03.8308612Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:25:03.8344548Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:25:03.8395122Z Entering 'third_party/flash-attention' 2025-12-04T09:25:03.8435473Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:25:03.8482022Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:25:03.8526173Z Entering 'third_party/flatbuffers' 2025-12-04T09:25:03.8565172Z Entering 'third_party/fmt' 2025-12-04T09:25:03.8610095Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:25:03.8644702Z Entering 'third_party/gloo' 2025-12-04T09:25:03.8690780Z Entering 'third_party/googletest' 2025-12-04T09:25:03.8728319Z Entering 'third_party/ideep' 2025-12-04T09:25:03.8771202Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:25:03.8813423Z Entering 'third_party/ittapi' 2025-12-04T09:25:03.8851238Z Entering 'third_party/kineto' 2025-12-04T09:25:03.8891614Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:25:03.8931491Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:25:03.8977890Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:25:03.9019386Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:25:03.9057974Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:25:03.9097685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:25:03.9137726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:25:03.9183700Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:25:03.9218627Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:25:03.9258975Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:25:03.9295483Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:25:03.9337411Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:03.9379447Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:03.9425670Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:25:03.9463389Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:25:03.9505036Z Entering 'third_party/kleidiai' 2025-12-04T09:25:03.9546478Z Entering 'third_party/mimalloc' 2025-12-04T09:25:03.9588087Z Entering 'third_party/nlohmann' 2025-12-04T09:25:03.9625120Z Entering 'third_party/onnx' 2025-12-04T09:25:03.9679924Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:25:03.9726165Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:25:03.9766886Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:25:03.9802587Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:25:03.9842191Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:25:03.9879180Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:25:03.9916948Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:25:03.9951606Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:25:03.9998285Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:25:04.0034528Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:04.0074756Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:04.0119664Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:25:04.0182444Z Entering 'third_party/pocketfft' 2025-12-04T09:25:04.0227578Z Entering 'third_party/protobuf' 2025-12-04T09:25:04.0267274Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:25:04.0306797Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:25:04.0346692Z Entering 'third_party/psimd' 2025-12-04T09:25:04.0389334Z Entering 'third_party/pthreadpool' 2025-12-04T09:25:04.0430996Z Entering 'third_party/pybind11' 2025-12-04T09:25:04.0478987Z Entering 'third_party/python-peachpy' 2025-12-04T09:25:04.0514815Z Entering 'third_party/sleef' 2025-12-04T09:25:04.0558193Z Entering 'third_party/tensorpipe' 2025-12-04T09:25:04.0599522Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:25:04.0641015Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:25:04.0685071Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:25:04.0721336Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:25:04.0759838Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:25:04.0820642Z ##[endgroup] 2025-12-04T09:25:04.0864476Z [command]/usr/bin/git log -1 --format=%H 2025-12-04T09:25:04.0888755Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:25:04.1007500Z ##[group]Run cd "${GITHUB_WORKSPACE}" 2025-12-04T09:25:04.1007750Z cd "${GITHUB_WORKSPACE}" 2025-12-04T09:25:04.1007949Z # Clean stale submodule dirs 2025-12-04T09:25:04.1008157Z if [ -z "${NO_SUDO}" ]; then 2025-12-04T09:25:04.1008397Z  sudo git submodule foreach --recursive git clean -ffdx 2025-12-04T09:25:04.1008620Z else 2025-12-04T09:25:04.1008830Z  git submodule foreach --recursive git clean -ffdx 2025-12-04T09:25:04.1009052Z fi 2025-12-04T09:25:04.1017170Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:04.1017406Z env: 2025-12-04T09:25:04.1017567Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:04.1017730Z NO_SUDO: true 2025-12-04T09:25:04.1017886Z ##[endgroup] 2025-12-04T09:25:04.1348592Z Entering 'android/libs/fbjni' 2025-12-04T09:25:04.1383448Z Entering 'third_party/FP16' 2025-12-04T09:25:04.1413034Z Entering 'third_party/FXdiv' 2025-12-04T09:25:04.1441545Z Entering 'third_party/NNPACK' 2025-12-04T09:25:04.1479238Z Entering 'third_party/NVTX' 2025-12-04T09:25:04.1516581Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:25:04.1551280Z Entering 'third_party/XNNPACK' 2025-12-04T09:25:04.1646439Z Entering 'third_party/aiter' 2025-12-04T09:25:04.1686321Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:25:04.1776476Z Entering 'third_party/benchmark' 2025-12-04T09:25:04.1811248Z Entering 'third_party/composable_kernel' 2025-12-04T09:25:04.1900994Z Entering 'third_party/cpp-httplib' 2025-12-04T09:25:04.1936583Z Entering 'third_party/cpuinfo' 2025-12-04T09:25:04.1973645Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:25:04.2006340Z Entering 'third_party/cutlass' 2025-12-04T09:25:04.2086051Z Entering 'third_party/fbgemm' 2025-12-04T09:25:04.2144908Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:25:04.2171347Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:25:04.2261369Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:25:04.2291841Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:25:04.2373634Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:25:04.2401460Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:25:04.2427692Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:25:04.2467887Z Entering 'third_party/flash-attention' 2025-12-04T09:25:04.2504488Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:25:04.2592956Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:25:04.2662270Z Entering 'third_party/flatbuffers' 2025-12-04T09:25:04.2723557Z Entering 'third_party/fmt' 2025-12-04T09:25:04.2756484Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:25:04.2790003Z Entering 'third_party/gloo' 2025-12-04T09:25:04.2820371Z Entering 'third_party/googletest' 2025-12-04T09:25:04.2853175Z Entering 'third_party/ideep' 2025-12-04T09:25:04.2879287Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:25:04.2956098Z Entering 'third_party/ittapi' 2025-12-04T09:25:04.2987701Z Entering 'third_party/kineto' 2025-12-04T09:25:04.3022379Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:25:04.3054001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:25:04.3093699Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:25:04.3124080Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:25:04.3152883Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:25:04.3181234Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:25:04.3214519Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:25:04.3244880Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:25:04.3277196Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:25:04.3314832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:25:04.3340357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:25:04.3370387Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:04.3414741Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:04.3454224Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:25:04.3485965Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:25:04.3513431Z Entering 'third_party/kleidiai' 2025-12-04T09:25:04.3552931Z Entering 'third_party/mimalloc' 2025-12-04T09:25:04.3586599Z Entering 'third_party/nlohmann' 2025-12-04T09:25:04.3623245Z Entering 'third_party/onnx' 2025-12-04T09:25:04.3843616Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:25:04.3880942Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:25:04.3928377Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:25:04.3962598Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:25:04.3994976Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:25:04.4023371Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:25:04.4062248Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:25:04.4092578Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:25:04.4127067Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:25:04.4155468Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:25:04.4200930Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:25:04.4232443Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:25:04.4417905Z Entering 'third_party/pocketfft' 2025-12-04T09:25:04.4443676Z Entering 'third_party/protobuf' 2025-12-04T09:25:04.4504336Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:25:04.4534440Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:25:04.4570096Z Entering 'third_party/psimd' 2025-12-04T09:25:04.4601532Z Entering 'third_party/pthreadpool' 2025-12-04T09:25:04.4631371Z Entering 'third_party/pybind11' 2025-12-04T09:25:04.4663301Z Entering 'third_party/python-peachpy' 2025-12-04T09:25:04.4699145Z Entering 'third_party/sleef' 2025-12-04T09:25:04.4731245Z Entering 'third_party/tensorpipe' 2025-12-04T09:25:04.4767348Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:25:04.4793342Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:25:04.4827840Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:25:04.4863023Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:25:04.4890134Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:25:04.5012927Z Prepare all required actions 2025-12-04T09:25:04.5013388Z Getting action download info 2025-12-04T09:25:04.6548130Z ##[group]Run ./.github/actions/setup-linux 2025-12-04T09:25:04.6548360Z env: 2025-12-04T09:25:04.6548538Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:04.6548714Z ##[endgroup] 2025-12-04T09:25:04.6579465Z ##[group]Run set -euo pipefail 2025-12-04T09:25:04.6579712Z set -euo pipefail 2025-12-04T09:25:04.6579908Z function get_ec2_metadata() { 2025-12-04T09:25:04.6580153Z  # Pulled from instance metadata endpoint for EC2 2025-12-04T09:25:04.6580536Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-12-04T09:25:04.6580869Z  category=$1 2025-12-04T09:25:04.6581104Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-12-04T09:25:04.6581375Z  runner_name_str=i-0c5ea43668efc70fe 2025-12-04T09:25:04.6581651Z  if [[ -f /.inarc ]]; then 2025-12-04T09:25:04.6581865Z  echo "ARC Runner, no info on ec2 metadata" 2025-12-04T09:25:04.6582103Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-12-04T09:25:04.6582380Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-12-04T09:25:04.6582628Z  else 2025-12-04T09:25:04.6583125Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-12-04T09:25:04.6583622Z  fi 2025-12-04T09:25:04.6583775Z } 2025-12-04T09:25:04.6583956Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-12-04T09:25:04.6584214Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-12-04T09:25:04.6584507Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-12-04T09:25:04.6584929Z echo "system info $(uname -a)" 2025-12-04T09:25:04.6590255Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:04.6590501Z env: 2025-12-04T09:25:04.6590663Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:04.6590831Z ##[endgroup] 2025-12-04T09:25:04.6746102Z ami-id: ami-08982f1c5bf93d976 2025-12-04T09:25:04.6845853Z instance-id: i-0c5ea43668efc70fe 2025-12-04T09:25:04.6936853Z instance-type: m7i-flex.8xlarge 2025-12-04T09:25:04.6946784Z system info Linux ip-10-0-39-147.ec2.internal 6.1.150-174.273.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 9 12:21:26 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-12-04T09:25:04.6972335Z ##[group]Run if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-12-04T09:25:04.6972660Z if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-12-04T09:25:04.6977094Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:04.6977334Z env: 2025-12-04T09:25:04.6977481Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:04.6977675Z ##[endgroup] 2025-12-04T09:25:04.7013912Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:25:04.7014472Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:25:04.7019008Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:04.7019255Z env: 2025-12-04T09:25:04.7019409Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:04.7019593Z ##[endgroup] 2025-12-04T09:25:04.7071261Z ##[group]Run if systemctl is-active --quiet docker; then 2025-12-04T09:25:04.7071534Z if systemctl is-active --quiet docker; then 2025-12-04T09:25:04.7071773Z  echo "Docker daemon is running..."; 2025-12-04T09:25:04.7071980Z else 2025-12-04T09:25:04.7072200Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-12-04T09:25:04.7072467Z fi 2025-12-04T09:25:04.7076526Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:04.7076764Z env: 2025-12-04T09:25:04.7076912Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:04.7077087Z ##[endgroup] 2025-12-04T09:25:04.7154064Z Docker daemon is running... 2025-12-04T09:25:04.7184262Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T09:25:04.7184486Z with: 2025-12-04T09:25:04.7184635Z shell: bash 2025-12-04T09:25:04.7184803Z timeout_minutes: 5 2025-12-04T09:25:04.7184982Z max_attempts: 3 2025-12-04T09:25:04.7185153Z retry_wait_seconds: 30 2025-12-04T09:25:04.7186583Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-12-04T09:25:04.7187966Z polling_interval_seconds: 1 2025-12-04T09:25:04.7188169Z warning_on_retry: true 2025-12-04T09:25:04.7188359Z continue_on_error: false 2025-12-04T09:25:04.7188537Z env: 2025-12-04T09:25:04.7188698Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:04.7188885Z AWS_RETRY_MODE: standard 2025-12-04T09:25:04.7189063Z AWS_MAX_ATTEMPTS: 5 2025-12-04T09:25:04.7189252Z AWS_DEFAULT_REGION: us-east-1 2025-12-04T09:25:04.7189475Z ##[endgroup] 2025-12-04T09:25:05.7124665Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:25:05.7125103Z Configure a credential helper to remove this warning. See 2025-12-04T09:25:05.7125500Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:25:05.7125927Z 2025-12-04T09:25:05.7126001Z Login Succeeded 2025-12-04T09:25:05.8468001Z Command completed after 1 attempt(s). 2025-12-04T09:25:05.8528548Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:25:05.8528909Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:25:05.8529245Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:25:05.8535928Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:05.8536182Z env: 2025-12-04T09:25:05.8536355Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:05.8536552Z ##[endgroup] 2025-12-04T09:25:05.8613418Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:25:05.8613831Z # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:25:05.8614098Z # shellcheck disable=SC2046 2025-12-04T09:25:05.8614322Z docker stop $(docker ps -q) || true 2025-12-04T09:25:05.8614546Z # Prune all of the docker images 2025-12-04T09:25:05.8614777Z docker system prune -af 2025-12-04T09:25:05.8619451Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:05.8619696Z env: 2025-12-04T09:25:05.8619849Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:05.8620030Z ##[endgroup] 2025-12-04T09:25:05.9090477Z "docker stop" requires at least 1 argument. 2025-12-04T09:25:05.9095422Z See 'docker stop --help'. 2025-12-04T09:25:05.9097767Z 2025-12-04T09:25:05.9098011Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-12-04T09:25:05.9098219Z 2025-12-04T09:25:05.9098314Z Stop one or more running containers 2025-12-04T09:25:05.9311373Z Total reclaimed space: 0B 2025-12-04T09:25:05.9448560Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-12-04T09:25:05.9448905Z with: 2025-12-04T09:25:05.9449480Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:05.9450133Z use-custom-docker-registry: true 2025-12-04T09:25:05.9450374Z docker-build-dir: .ci/docker 2025-12-04T09:25:05.9450600Z docker-build-script: ./build.sh 2025-12-04T09:25:05.9450814Z working-directory: . 2025-12-04T09:25:05.9451078Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:25:05.9451361Z force-push: false 2025-12-04T09:25:05.9451530Z env: 2025-12-04T09:25:05.9451699Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:05.9451896Z ##[endgroup] 2025-12-04T09:25:05.9466653Z ##[group]Run set -ex 2025-12-04T09:25:05.9466887Z set -ex 2025-12-04T09:25:05.9467058Z  2025-12-04T09:25:05.9467380Z # If the docker build directory or the build script doesn't exist, the action will 2025-12-04T09:25:05.9467808Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-12-04T09:25:05.9468192Z # job could then download the pre-built image as usual 2025-12-04T09:25:05.9468654Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-12-04T09:25:05.9469075Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-12-04T09:25:05.9469303Z else 2025-12-04T09:25:05.9469517Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-12-04T09:25:05.9469823Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:25:05.9470085Z  2025-12-04T09:25:05.9470464Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-12-04T09:25:05.9470877Z  exit 0 2025-12-04T09:25:05.9471046Z fi 2025-12-04T09:25:05.9471198Z  2025-12-04T09:25:05.9471441Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-12-04T09:25:05.9471839Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-12-04T09:25:05.9472291Z  # use it as it is, but first let's extract the tag 2025-12-04T09:25:05.9472619Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-12-04T09:25:05.9472981Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:25:05.9473309Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:25:05.9473596Z else 2025-12-04T09:25:05.9473798Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-12-04T09:25:05.9474074Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-12-04T09:25:05.9474343Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-12-04T09:25:05.9474582Z  fi 2025-12-04T09:25:05.9474894Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-12-04T09:25:05.9475296Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:25:05.9475707Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:25:05.9476172Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:25:05.9476458Z fi 2025-12-04T09:25:05.9481501Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:05.9481776Z env: 2025-12-04T09:25:05.9481952Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:05.9482154Z REPO_NAME: pytorch 2025-12-04T09:25:05.9482808Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:05.9483369Z DOCKER_BUILD_DIR: .ci/docker 2025-12-04T09:25:05.9483571Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-12-04T09:25:05.9483821Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:25:05.9484083Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-12-04T09:25:05.9484289Z CUSTOM_TAG_PREFIX: 2025-12-04T09:25:05.9484464Z ##[endgroup] 2025-12-04T09:25:05.9510585Z + [[ -d .ci/docker ]] 2025-12-04T09:25:05.9510849Z + [[ -f .ci/docker/./build.sh ]] 2025-12-04T09:25:05.9511089Z + [[ true == \t\r\u\e ]] 2025-12-04T09:25:05.9511306Z + echo skip=false 2025-12-04T09:25:05.9512102Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-12-04T09:25:05.9517524Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:05.9518131Z ++ awk -F '[:,]' '{print $2}' 2025-12-04T09:25:05.9540113Z + DOCKER_TAG=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:05.9540897Z + echo docker-tag=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:05.9541689Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:05.9560574Z ##[group]Run set +e 2025-12-04T09:25:05.9560827Z set +e 2025-12-04T09:25:05.9561002Z set -x 2025-12-04T09:25:05.9561171Z  2025-12-04T09:25:05.9561336Z login() { 2025-12-04T09:25:05.9561667Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-12-04T09:25:05.9562033Z } 2025-12-04T09:25:05.9562183Z  2025-12-04T09:25:05.9562325Z retry () { 2025-12-04T09:25:05.9562515Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-12-04T09:25:05.9562722Z } 2025-12-04T09:25:05.9562863Z  2025-12-04T09:25:05.9563024Z retry login "${DOCKER_REGISTRY}" 2025-12-04T09:25:05.9563344Z  2025-12-04T09:25:05.9563501Z START_TIME=$(date +%s) 2025-12-04T09:25:05.9563697Z # Wait up to 120 minutes 2025-12-04T09:25:05.9563949Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-12-04T09:25:05.9564263Z  # Check if image already exists, if it does then skip building it 2025-12-04T09:25:05.9564571Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-12-04T09:25:05.9564810Z  exit 0 2025-12-04T09:25:05.9564976Z  fi 2025-12-04T09:25:05.9565120Z  2025-12-04T09:25:05.9565372Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-12-04T09:25:05.9565773Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-12-04T09:25:05.9566171Z  # latter, it will wait for the Docker images to become available before continuing 2025-12-04T09:25:05.9566498Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-12-04T09:25:05.9566764Z  # It's a Docker build job, let's build the image 2025-12-04T09:25:05.9566984Z  break 2025-12-04T09:25:05.9567147Z  else 2025-12-04T09:25:05.9567371Z  # It's a regular build job, wait for the image to become available 2025-12-04T09:25:05.9567629Z  sleep 300 2025-12-04T09:25:05.9567797Z  fi 2025-12-04T09:25:05.9567945Z done 2025-12-04T09:25:05.9568096Z  2025-12-04T09:25:05.9568328Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-12-04T09:25:05.9568763Z # be empty. The default action would be to continue rebuild the image 2025-12-04T09:25:05.9569087Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-12-04T09:25:05.9569374Z  # if we're on the base branch then use the parent commit 2025-12-04T09:25:05.9569637Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-12-04T09:25:05.9569835Z else 2025-12-04T09:25:05.9570056Z  # otherwise we're on a PR, so use the most recent base commit 2025-12-04T09:25:05.9570357Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-12-04T09:25:05.9570577Z fi 2025-12-04T09:25:05.9570734Z  2025-12-04T09:25:05.9570898Z if [[ -z "${MERGE_BASE}" ]]; then 2025-12-04T09:25:05.9571125Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-12-04T09:25:05.9571326Z  2025-12-04T09:25:05.9571607Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-12-04T09:25:05.9571925Z  exit 0 2025-12-04T09:25:05.9572071Z fi 2025-12-04T09:25:05.9572216Z  2025-12-04T09:25:05.9572418Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-12-04T09:25:05.9572809Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-12-04T09:25:05.9573139Z  exit 1 2025-12-04T09:25:05.9573291Z fi 2025-12-04T09:25:05.9573434Z  2025-12-04T09:25:05.9573658Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-12-04T09:25:05.9574045Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-12-04T09:25:05.9574387Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-12-04T09:25:05.9574787Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-12-04T09:25:05.9575218Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-12-04T09:25:05.9575516Z fi 2025-12-04T09:25:05.9575661Z  2025-12-04T09:25:05.9575828Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-12-04T09:25:05.9580886Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:05.9581141Z env: 2025-12-04T09:25:05.9581291Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:05.9581483Z DOCKER_BUILD_DIR: .ci/docker 2025-12-04T09:25:05.9581720Z BASE_REVISION: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:25:05.9582298Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:05.9583004Z DOCKER_TAG: pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:05.9583442Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:25:05.9583689Z DOCKER_PUSH: 2025-12-04T09:25:05.9583842Z ##[endgroup] 2025-12-04T09:25:05.9607309Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:25:05.9612073Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:25:05.9612474Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:25:05.9612799Z + aws ecr get-login-password --region us-east-1 2025-12-04T09:25:06.3795546Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:25:06.3795918Z Login Succeeded 2025-12-04T09:25:06.3796397Z Configure a credential helper to remove this warning. See 2025-12-04T09:25:06.3796782Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:25:06.3797027Z 2025-12-04T09:25:06.3819711Z ++ date +%s 2025-12-04T09:25:06.3828626Z + START_TIME=1764840306 2025-12-04T09:25:06.3833097Z ++ date +%s 2025-12-04T09:25:06.3840983Z + [[ 1764833106 -lt 1764840306 ]] 2025-12-04T09:25:06.3841605Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:06.5905466Z { 2025-12-04T09:25:06.5905721Z "schemaVersion": 2, 2025-12-04T09:25:06.5906083Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-12-04T09:25:06.5906385Z "config": { 2025-12-04T09:25:06.5906634Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-12-04T09:25:06.5906897Z "size": 33581, 2025-12-04T09:25:06.5907166Z "digest": "sha256:40bd34edc3497e05e2150804a15313235b6426bcc4b8c19c03c3daf5524be913" 2025-12-04T09:25:06.5907462Z }, 2025-12-04T09:25:06.5907608Z "layers": [ 2025-12-04T09:25:06.5907751Z { 2025-12-04T09:25:06.5907985Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5908338Z "size": 30447951, 2025-12-04T09:25:06.5908628Z "digest": "sha256:63e5bc7682b85ae57a1221210f64d62e7a90b0a30f19af4ca734b8242ae49d63" 2025-12-04T09:25:06.5908923Z }, 2025-12-04T09:25:06.5909062Z { 2025-12-04T09:25:06.5909280Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5909588Z "size": 1554, 2025-12-04T09:25:06.5909907Z "digest": "sha256:6dc15eca51381c13be16385052dd2378ab1dce5fb77f7e7bceab34ed72e6e0e5" 2025-12-04T09:25:06.5910245Z }, 2025-12-04T09:25:06.5910407Z { 2025-12-04T09:25:06.5910622Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5910904Z "size": 313274856, 2025-12-04T09:25:06.5911172Z "digest": "sha256:459f4df18f07c473cb662e8033f9dab9f68c9ac71608767ad055999fae6b31e2" 2025-12-04T09:25:06.5911462Z }, 2025-12-04T09:25:06.5911597Z { 2025-12-04T09:25:06.5911807Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5912066Z "size": 787, 2025-12-04T09:25:06.5912339Z "digest": "sha256:82108541691974f1cc469d0ec89a048191b8de4dc36bf9a0184a4bd50120b69d" 2025-12-04T09:25:06.5912616Z }, 2025-12-04T09:25:06.5912739Z { 2025-12-04T09:25:06.5912945Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5913196Z "size": 106, 2025-12-04T09:25:06.5913443Z "digest": "sha256:3738646b3d92a3f1b6529ba588c00216c306c997ad77730f207c7107c2e5aff3" 2025-12-04T09:25:06.5914053Z }, 2025-12-04T09:25:06.5914186Z { 2025-12-04T09:25:06.5914393Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5914648Z "size": 704, 2025-12-04T09:25:06.5914903Z "digest": "sha256:aa9a10a37b3945f69568695fa4a9321333654a69dd44c55eff4e26bf9bffcad1" 2025-12-04T09:25:06.5915179Z }, 2025-12-04T09:25:06.5915314Z { 2025-12-04T09:25:06.5915527Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5915772Z "size": 1218, 2025-12-04T09:25:06.5916035Z "digest": "sha256:55bac08f3e18897f61d3e7f28ada6547b05b3c36f559a882656834d43de61478" 2025-12-04T09:25:06.5916324Z }, 2025-12-04T09:25:06.5916456Z { 2025-12-04T09:25:06.5916658Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5916928Z "size": 485, 2025-12-04T09:25:06.5917214Z "digest": "sha256:acf6468f6aadc3613e3b8fe73c016ce8d6f47f01e0cfee3a1015d0335c24d691" 2025-12-04T09:25:06.5917527Z }, 2025-12-04T09:25:06.5917666Z { 2025-12-04T09:25:06.5917886Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5918161Z "size": 110362453, 2025-12-04T09:25:06.5918454Z "digest": "sha256:c0c31a0e69b798875a8d2c05d01a6861b2fe623e1970bf7faf822e7df6691d2c" 2025-12-04T09:25:06.5918764Z }, 2025-12-04T09:25:06.5918894Z { 2025-12-04T09:25:06.5919119Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5919396Z "size": 4961, 2025-12-04T09:25:06.5919678Z "digest": "sha256:0f7d0dc70d2d944cc1c8a592f051d3f28d2b062c79166713fb7cc9225b4c97c0" 2025-12-04T09:25:06.5919977Z }, 2025-12-04T09:25:06.5920224Z { 2025-12-04T09:25:06.5920749Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5921033Z "size": 1756, 2025-12-04T09:25:06.5921324Z "digest": "sha256:58c4d15d3bc18a789eb0d1a13f68df0b1d6cea66697fe63adf11b53fec74ba18" 2025-12-04T09:25:06.5921645Z }, 2025-12-04T09:25:06.5921782Z { 2025-12-04T09:25:06.5922005Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5922289Z "size": 724, 2025-12-04T09:25:06.5922574Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T09:25:06.5922871Z }, 2025-12-04T09:25:06.5923056Z { 2025-12-04T09:25:06.5923314Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5923647Z "size": 543, 2025-12-04T09:25:06.5923939Z "digest": "sha256:d2472a74103d877e6532d6409a2ac9d54db6672b5d3002be011fdbbd8d360c49" 2025-12-04T09:25:06.5924231Z }, 2025-12-04T09:25:06.5924357Z { 2025-12-04T09:25:06.5924586Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5924858Z "size": 3353192867, 2025-12-04T09:25:06.5925264Z "digest": "sha256:d1fba72936888ea8f8e4c7ad015bfcf3285017b95f3d72795d5b6a0b0ded36fc" 2025-12-04T09:25:06.5925557Z }, 2025-12-04T09:25:06.5925691Z { 2025-12-04T09:25:06.5925907Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5926158Z "size": 32, 2025-12-04T09:25:06.5926434Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.5926719Z }, 2025-12-04T09:25:06.5926841Z { 2025-12-04T09:25:06.5927050Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5927302Z "size": 397, 2025-12-04T09:25:06.5927555Z "digest": "sha256:6a9b03ce41a77be16366fba2c6ca0aabefe42ee4879ab5d088b312970e830770" 2025-12-04T09:25:06.5927840Z }, 2025-12-04T09:25:06.5927967Z { 2025-12-04T09:25:06.5928177Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5928430Z "size": 237420, 2025-12-04T09:25:06.5928693Z "digest": "sha256:3519ac15be7978a14201f5d431b6472ed30382cde8628c6401c9db98ccd1a871" 2025-12-04T09:25:06.5928975Z }, 2025-12-04T09:25:06.5929103Z { 2025-12-04T09:25:06.5929343Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5929661Z "size": 231, 2025-12-04T09:25:06.5929914Z "digest": "sha256:dabb51b819a06823e5a1c8d4f9e29b689f38b19ee1e90da40075bd1593b8d0fb" 2025-12-04T09:25:06.5930205Z }, 2025-12-04T09:25:06.5930342Z { 2025-12-04T09:25:06.5930550Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5930812Z "size": 2967944, 2025-12-04T09:25:06.5931083Z "digest": "sha256:8f36833a24d02d47bf18bdc573adbb45afb8f5f06886da9bd671a1a33e3007bd" 2025-12-04T09:25:06.5931369Z }, 2025-12-04T09:25:06.5931505Z { 2025-12-04T09:25:06.5931720Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5931979Z "size": 1472, 2025-12-04T09:25:06.5932244Z "digest": "sha256:ab53c5a853a52de13f11fbb026aaf469754adbe52b02e96f4fbdc05916585244" 2025-12-04T09:25:06.5932537Z }, 2025-12-04T09:25:06.5932670Z { 2025-12-04T09:25:06.5932876Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5933133Z "size": 482, 2025-12-04T09:25:06.5933391Z "digest": "sha256:025a0e5e6ac19cb2d27c9423a5d019a34ea7d4004d426d6b890693ccfe9f54d3" 2025-12-04T09:25:06.5933673Z }, 2025-12-04T09:25:06.5933805Z { 2025-12-04T09:25:06.5934017Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5934271Z "size": 200, 2025-12-04T09:25:06.5934553Z "digest": "sha256:ce3394c8f2109f4538f6e7f2c39f685d0ba028aacac33ddacbb8ef9fd349702e" 2025-12-04T09:25:06.5934999Z }, 2025-12-04T09:25:06.5935145Z { 2025-12-04T09:25:06.5935385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5935660Z "size": 608, 2025-12-04T09:25:06.5935985Z "digest": "sha256:a5c3888c3a0c5812efbfe307d18de826363900fa8d8097c1ee84a72630aa067b" 2025-12-04T09:25:06.5936292Z }, 2025-12-04T09:25:06.5936433Z { 2025-12-04T09:25:06.5936648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5936917Z "size": 225, 2025-12-04T09:25:06.5937199Z "digest": "sha256:ed902a3a4e3b916aa869b612baf9cd36745b671b6781d19e3b4c3fadc2f513f5" 2025-12-04T09:25:06.5937501Z }, 2025-12-04T09:25:06.5937636Z { 2025-12-04T09:25:06.5937847Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5938093Z "size": 829, 2025-12-04T09:25:06.5938356Z "digest": "sha256:b4e1efca22beb475459a14f4a571a2512c2b1ed4b24e66bd3f82f1681dc5b1bc" 2025-12-04T09:25:06.5938649Z }, 2025-12-04T09:25:06.5938776Z { 2025-12-04T09:25:06.5938986Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5939239Z "size": 32, 2025-12-04T09:25:06.5939501Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.5939786Z }, 2025-12-04T09:25:06.5939922Z { 2025-12-04T09:25:06.5940139Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5940396Z "size": 104, 2025-12-04T09:25:06.5940674Z "digest": "sha256:79fcfd297d9fcfe2810595fc2c6ba503293b59046bc192bb0620a64d9bdff778" 2025-12-04T09:25:06.5940983Z }, 2025-12-04T09:25:06.5941114Z { 2025-12-04T09:25:06.5941335Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5941604Z "size": 1495, 2025-12-04T09:25:06.5941874Z "digest": "sha256:7bd89c134b49f83cfb0725df3615556b60af380b932186c5abd001232b4597cf" 2025-12-04T09:25:06.5942173Z }, 2025-12-04T09:25:06.5942313Z { 2025-12-04T09:25:06.5942528Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5942798Z "size": 458790783, 2025-12-04T09:25:06.5943092Z "digest": "sha256:b7d40b4fd1b9375cfd45d5c593ec0cccc21c80d667ea79f4effb09f04b0705b0" 2025-12-04T09:25:06.5943400Z }, 2025-12-04T09:25:06.5943532Z { 2025-12-04T09:25:06.5943751Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5944020Z "size": 163, 2025-12-04T09:25:06.5944293Z "digest": "sha256:dd1cdd87320d29d4ca09686ab00b76a396efcc6f3ea6d0bfd1f7922e46336ca6" 2025-12-04T09:25:06.5944650Z }, 2025-12-04T09:25:06.5944791Z { 2025-12-04T09:25:06.5945005Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5945275Z "size": 347, 2025-12-04T09:25:06.5945551Z "digest": "sha256:c21a6e1cd03a718ebfb927ab114ebbbee0b57a93908d0265d24b02cf34c0fee3" 2025-12-04T09:25:06.5945855Z }, 2025-12-04T09:25:06.5945995Z { 2025-12-04T09:25:06.5946213Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5946490Z "size": 32, 2025-12-04T09:25:06.5946770Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.5947324Z }, 2025-12-04T09:25:06.5947483Z { 2025-12-04T09:25:06.5947703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5947981Z "size": 106, 2025-12-04T09:25:06.5948275Z "digest": "sha256:0aa3ea0b275426eac8eff6b26ba17c168ae70d1baa5a7845e61d41bd01a9ff43" 2025-12-04T09:25:06.5948583Z }, 2025-12-04T09:25:06.5948726Z { 2025-12-04T09:25:06.5948950Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5949213Z "size": 426, 2025-12-04T09:25:06.5949494Z "digest": "sha256:74e0bdbb05d31425d7a2f40ff0cb2423abaf90f1ed56c4b69f43e8d92c569379" 2025-12-04T09:25:06.5949814Z }, 2025-12-04T09:25:06.5949947Z { 2025-12-04T09:25:06.5950172Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5950452Z "size": 19309400, 2025-12-04T09:25:06.5950745Z "digest": "sha256:ba3aee7dcf03efbd1e7b1611c8f65f385c94b2572d15c099a1535f27b72509f7" 2025-12-04T09:25:06.5951055Z }, 2025-12-04T09:25:06.5951279Z { 2025-12-04T09:25:06.5951508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5951773Z "size": 108, 2025-12-04T09:25:06.5952049Z "digest": "sha256:8c8d59b8759d40399b07b019a46fc05b9ccabe2b9276f49eea203fd6908ff334" 2025-12-04T09:25:06.5952361Z }, 2025-12-04T09:25:06.5952498Z { 2025-12-04T09:25:06.5952718Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5952992Z "size": 827, 2025-12-04T09:25:06.5953262Z "digest": "sha256:3c960865867b66327e316bf682adabd1867dd9a41cc0aee1507ee8e35fa614a9" 2025-12-04T09:25:06.5953566Z }, 2025-12-04T09:25:06.5953707Z { 2025-12-04T09:25:06.5953922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5954189Z "size": 724, 2025-12-04T09:25:06.5954472Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T09:25:06.5954782Z }, 2025-12-04T09:25:06.5954913Z { 2025-12-04T09:25:06.5955138Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5955406Z "size": 149, 2025-12-04T09:25:06.5955675Z "digest": "sha256:074db79e3832af094ddad6e5c5649347b62e8ba0365229feced3ed0263a0c611" 2025-12-04T09:25:06.5955974Z }, 2025-12-04T09:25:06.5956108Z { 2025-12-04T09:25:06.5956307Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5956562Z "size": 137, 2025-12-04T09:25:06.5956820Z "digest": "sha256:226748f4ff23ba2aef33619c3a639fb44865d5f4515c4b45d4196f4e0e77b187" 2025-12-04T09:25:06.5957101Z }, 2025-12-04T09:25:06.5957238Z { 2025-12-04T09:25:06.5957462Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5957732Z "size": 140, 2025-12-04T09:25:06.5957997Z "digest": "sha256:75d817336f1df7579f5b4be29625c53d594cb78c776f5454cc69438bb1bb0dc9" 2025-12-04T09:25:06.5958296Z }, 2025-12-04T09:25:06.5958432Z { 2025-12-04T09:25:06.5958654Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5958915Z "size": 18890247472, 2025-12-04T09:25:06.5959188Z "digest": "sha256:850870a17b94a4d7c5e55b287cc0b1ddd0982845e11f055765dc358277cdfc54" 2025-12-04T09:25:06.5959465Z }, 2025-12-04T09:25:06.5959598Z { 2025-12-04T09:25:06.5959819Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5960148Z "size": 222, 2025-12-04T09:25:06.5960492Z "digest": "sha256:e6225129924f0a66acaf35efd792901351da58116629962669d9c3acaadbab6e" 2025-12-04T09:25:06.5960801Z }, 2025-12-04T09:25:06.5960935Z { 2025-12-04T09:25:06.5961160Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5961433Z "size": 255, 2025-12-04T09:25:06.5961719Z "digest": "sha256:5cd6b9ec5c3c4a1cf2e2e7a1794aaae173eb5a291b9899bb733e5f36d866fab0" 2025-12-04T09:25:06.5962021Z }, 2025-12-04T09:25:06.5962153Z { 2025-12-04T09:25:06.5962376Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5962621Z "size": 32, 2025-12-04T09:25:06.5962880Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.5963165Z }, 2025-12-04T09:25:06.5963287Z { 2025-12-04T09:25:06.5963489Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5963747Z "size": 106, 2025-12-04T09:25:06.5964002Z "digest": "sha256:613d2b1f3db875d1d695081b44aefbf94eeb7815d794b28dd17ec623d3f9dad5" 2025-12-04T09:25:06.5964342Z }, 2025-12-04T09:25:06.5964470Z { 2025-12-04T09:25:06.5964668Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5964919Z "size": 312293443, 2025-12-04T09:25:06.5965180Z "digest": "sha256:388169fffe8a06396f4e8d6fc51d07a83e08f04836995ae28c2e13960cb89fab" 2025-12-04T09:25:06.5965461Z }, 2025-12-04T09:25:06.5965582Z { 2025-12-04T09:25:06.5965784Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5966037Z "size": 6628443345, 2025-12-04T09:25:06.5966366Z "digest": "sha256:6ca498e78b7b821a815bc7c14f057d52f7d9ce8dda8bca0945e6f522c7d208e1" 2025-12-04T09:25:06.5966648Z }, 2025-12-04T09:25:06.5966776Z { 2025-12-04T09:25:06.5966973Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5967227Z "size": 129, 2025-12-04T09:25:06.5967491Z "digest": "sha256:952b753ec7be39e1967f95afb87a880712b6425610adcda2afe3b2211cc8de12" 2025-12-04T09:25:06.5967771Z }, 2025-12-04T09:25:06.5967905Z { 2025-12-04T09:25:06.5968119Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5968367Z "size": 880, 2025-12-04T09:25:06.5968622Z "digest": "sha256:5b6f6de4bdd1c96aa8704c326fb27fa7465ebf388f0748ee56d9f521bb6b6697" 2025-12-04T09:25:06.5968911Z }, 2025-12-04T09:25:06.5969047Z { 2025-12-04T09:25:06.5969253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5969509Z "size": 724, 2025-12-04T09:25:06.5969788Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T09:25:06.5970064Z }, 2025-12-04T09:25:06.5970195Z { 2025-12-04T09:25:06.5970399Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5970645Z "size": 139, 2025-12-04T09:25:06.5970912Z "digest": "sha256:fa23d9952f3cea91af7b5ffce93e64d90d98aefb838aa8a1fe4a6b40cd0eab91" 2025-12-04T09:25:06.5971221Z }, 2025-12-04T09:25:06.5971344Z { 2025-12-04T09:25:06.5971551Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5971802Z "size": 32, 2025-12-04T09:25:06.5972064Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.5972341Z }, 2025-12-04T09:25:06.5972470Z { 2025-12-04T09:25:06.5972673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5972914Z "size": 161, 2025-12-04T09:25:06.5973182Z "digest": "sha256:c2aa9ea4d09ac3edb41e48cdc892fafe72a5cddc98ebdf67d978c6f8d63cd7d2" 2025-12-04T09:25:06.5973474Z }, 2025-12-04T09:25:06.5973594Z { 2025-12-04T09:25:06.5973797Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5974044Z "size": 1011, 2025-12-04T09:25:06.5974298Z "digest": "sha256:a9bbc9e426d367e15c7f0c1faae6417526e219fa8188ae17b9e6c2c3b8083bdc" 2025-12-04T09:25:06.5974626Z }, 2025-12-04T09:25:06.5974755Z { 2025-12-04T09:25:06.5974953Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5975197Z "size": 724, 2025-12-04T09:25:06.5975486Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T09:25:06.5975766Z }, 2025-12-04T09:25:06.5975887Z { 2025-12-04T09:25:06.5976091Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5976340Z "size": 135, 2025-12-04T09:25:06.5976588Z "digest": "sha256:82f2371f6dc2c2797baa60e672fd78bb04909cd0248c52cae058a7fd8f215a4f" 2025-12-04T09:25:06.5976869Z }, 2025-12-04T09:25:06.5977005Z { 2025-12-04T09:25:06.5977203Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5977449Z "size": 32, 2025-12-04T09:25:06.5977703Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.5977978Z }, 2025-12-04T09:25:06.5978112Z { 2025-12-04T09:25:06.5978320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5978567Z "size": 158, 2025-12-04T09:25:06.5978815Z "digest": "sha256:c87301921afac5ba5745add04b44f57bf31c551bf48a2574d4424dab64c25ce2" 2025-12-04T09:25:06.5979094Z }, 2025-12-04T09:25:06.5979226Z { 2025-12-04T09:25:06.5979425Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5979669Z "size": 603, 2025-12-04T09:25:06.5979927Z "digest": "sha256:384d566a822f4defa0ecf9c7d37b6d7a5fc5eaab415fe1d3cc02b190ddc19e71" 2025-12-04T09:25:06.5980219Z }, 2025-12-04T09:25:06.5980350Z { 2025-12-04T09:25:06.5980594Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5980836Z "size": 724, 2025-12-04T09:25:06.5981092Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T09:25:06.5981380Z }, 2025-12-04T09:25:06.5981500Z { 2025-12-04T09:25:06.5981708Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5981955Z "size": 155, 2025-12-04T09:25:06.5982219Z "digest": "sha256:32ddcdd5dd6920acb12ca95671a164346308ec7743afbe8a333142733c42bafd" 2025-12-04T09:25:06.5982482Z }, 2025-12-04T09:25:06.5982610Z { 2025-12-04T09:25:06.5982810Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5983044Z "size": 32, 2025-12-04T09:25:06.5983292Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.5983563Z }, 2025-12-04T09:25:06.5983681Z { 2025-12-04T09:25:06.5983881Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5984122Z "size": 188, 2025-12-04T09:25:06.5984370Z "digest": "sha256:a116a32ceaa15bbd55b0f4281fb0040de221c62e29659b23dd71ecd2382a969a" 2025-12-04T09:25:06.5984651Z }, 2025-12-04T09:25:06.5984788Z { 2025-12-04T09:25:06.5984981Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.5985228Z "size": 1371, 2025-12-04T09:25:06.6015838Z "digest": "sha256:f6fbbbe6067e383cadb530c3f53b113e9826843fb2348ea1f9a000a4790fa822" 2025-12-04T09:25:06.6016273Z }, 2025-12-04T09:25:06.6016421Z { 2025-12-04T09:25:06.6016668Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6016950Z "size": 32, 2025-12-04T09:25:06.6017232Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.6017536Z }, 2025-12-04T09:25:06.6017683Z { 2025-12-04T09:25:06.6017902Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6018226Z "size": 137, 2025-12-04T09:25:06.6018541Z "digest": "sha256:648018658875ac3356b277abe3d98df9a8b7a0345f131598c07f5e9f9318b740" 2025-12-04T09:25:06.6018834Z }, 2025-12-04T09:25:06.6018970Z { 2025-12-04T09:25:06.6019201Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6019629Z "size": 528, 2025-12-04T09:25:06.6019910Z "digest": "sha256:5e2f7eac20ad5128eebe0ba7dbd08111d28e7f65c26c7fea7cd6dc7a2c0725b9" 2025-12-04T09:25:06.6020216Z }, 2025-12-04T09:25:06.6020354Z { 2025-12-04T09:25:06.6020563Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6020821Z "size": 32, 2025-12-04T09:25:06.6021083Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.6021371Z }, 2025-12-04T09:25:06.6021511Z { 2025-12-04T09:25:06.6021723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6021972Z "size": 104, 2025-12-04T09:25:06.6022243Z "digest": "sha256:0d1243af7593a042ff443492a4cac70f817dc8c56ac45a0c138d102f40c1cc07" 2025-12-04T09:25:06.6022529Z }, 2025-12-04T09:25:06.6022669Z { 2025-12-04T09:25:06.6022878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6023139Z "size": 435, 2025-12-04T09:25:06.6023413Z "digest": "sha256:1f6d5b941ea1f6f07c190f5ca1d4a5172e4d696ad597ae58dcd37b297f60b556" 2025-12-04T09:25:06.6023700Z }, 2025-12-04T09:25:06.6023837Z { 2025-12-04T09:25:06.6024053Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6024302Z "size": 32, 2025-12-04T09:25:06.6024568Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.6024863Z }, 2025-12-04T09:25:06.6024993Z { 2025-12-04T09:25:06.6025209Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6025469Z "size": 107, 2025-12-04T09:25:06.6025801Z "digest": "sha256:4444f70d73cefec9f161242e9def76078af5abc7a3ad3e5d6e7d4f1efed07939" 2025-12-04T09:25:06.6026107Z }, 2025-12-04T09:25:06.6026248Z { 2025-12-04T09:25:06.6026473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6026736Z "size": 1896, 2025-12-04T09:25:06.6027017Z "digest": "sha256:bf45550ddbad2524852930466f963e432429f2b86f7f6ccf0415fc56980004ce" 2025-12-04T09:25:06.6027311Z }, 2025-12-04T09:25:06.6027445Z { 2025-12-04T09:25:06.6027662Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6027928Z "size": 245588924, 2025-12-04T09:25:06.6028204Z "digest": "sha256:30fff4f4bad325a3a2d91463ca6702c411392e5b0646a5cd4f47fcae56c55639" 2025-12-04T09:25:06.6049143Z }, 2025-12-04T09:25:06.6049318Z { 2025-12-04T09:25:06.6049535Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6049794Z "size": 106, 2025-12-04T09:25:06.6050073Z "digest": "sha256:cca4db1ba155a3afd1fd645cd3223e9a4de9c785f8559387f21da187672d9e9e" 2025-12-04T09:25:06.6050368Z }, 2025-12-04T09:25:06.6050496Z { 2025-12-04T09:25:06.6050708Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6050959Z "size": 165, 2025-12-04T09:25:06.6051224Z "digest": "sha256:f0ffa379f4eba2f3ddc323f45f226f39aed5a7e93880884a97bd4d6a91864dc1" 2025-12-04T09:25:06.6051518Z }, 2025-12-04T09:25:06.6051643Z { 2025-12-04T09:25:06.6051847Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6052100Z "size": 7942, 2025-12-04T09:25:06.6052355Z "digest": "sha256:df645e678f95db120f96e4c4c66400d315ed9214d9085216b417348cf682d43a" 2025-12-04T09:25:06.6052628Z }, 2025-12-04T09:25:06.6052751Z { 2025-12-04T09:25:06.6052954Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6053204Z "size": 8076, 2025-12-04T09:25:06.6053462Z "digest": "sha256:52ad6c17d10308710fb2dc855f911246b7a369ccf5c3142cb3ad7fb182f708b8" 2025-12-04T09:25:06.6053741Z }, 2025-12-04T09:25:06.6053865Z { 2025-12-04T09:25:06.6054069Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6054316Z "size": 302, 2025-12-04T09:25:06.6054569Z "digest": "sha256:34408374e32c70dbcfd7e0516edaa8698c25e6e3dbcd920978f2b3a801bbe659" 2025-12-04T09:25:06.6054969Z }, 2025-12-04T09:25:06.6055091Z { 2025-12-04T09:25:06.6055293Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6055541Z "size": 32, 2025-12-04T09:25:06.6055793Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.6056072Z }, 2025-12-04T09:25:06.6056193Z { 2025-12-04T09:25:06.6056398Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6056643Z "size": 108, 2025-12-04T09:25:06.6056897Z "digest": "sha256:27803b661d9e2cb1ce49c0a100ce824fbcbdd1f3a3ceda4ed5affbcb07ac3be4" 2025-12-04T09:25:06.6057182Z }, 2025-12-04T09:25:06.6057303Z { 2025-12-04T09:25:06.6057508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6057753Z "size": 54145699, 2025-12-04T09:25:06.6058017Z "digest": "sha256:07dafc893dea952d7677124b42e90b29b167fabe81e8a9d8b3fb4b0aa11b66ec" 2025-12-04T09:25:06.6058295Z }, 2025-12-04T09:25:06.6058420Z { 2025-12-04T09:25:06.6058618Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:25:06.6058861Z "size": 32, 2025-12-04T09:25:06.6059108Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:25:06.6059388Z } 2025-12-04T09:25:06.6059511Z ] 2025-12-04T09:25:06.6059638Z } 2025-12-04T09:25:06.6059787Z + exit 0 2025-12-04T09:25:06.6081421Z ##[group]Run set -eux 2025-12-04T09:25:06.6081907Z set -eux 2025-12-04T09:25:06.6082174Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have 2025-12-04T09:25:06.6082935Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true 2025-12-04T09:25:06.6087995Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:06.6088237Z env: 2025-12-04T09:25:06.6088408Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:06.6088584Z ##[endgroup] 2025-12-04T09:25:06.6113193Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-12-04T09:25:06.6116104Z + jq --raw-output .SecretString 2025-12-04T09:25:06.6116410Z + jq -r .docker_hub_readonly_token 2025-12-04T09:25:06.6116766Z + docker login --username pytorchbot --password-stdin 2025-12-04T09:25:07.0682953Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:25:07.0683410Z Configure a credential helper to remove this warning. See 2025-12-04T09:25:07.0683858Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:25:07.0684147Z 2025-12-04T09:25:07.0686718Z Login Succeeded 2025-12-04T09:25:07.0771967Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-12-04T09:25:07.0772238Z tag=${ECR_DOCKER_IMAGE##*:} 2025-12-04T09:25:07.0772507Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-12-04T09:25:07.0777212Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:07.0777460Z env: 2025-12-04T09:25:07.0777625Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:07.0778180Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:07.0778707Z ##[endgroup] 2025-12-04T09:25:07.0803665Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:07.0834984Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-12-04T09:25:07.0835315Z with: 2025-12-04T09:25:07.0835874Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:07.0836533Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:25:07.0836843Z env: 2025-12-04T09:25:07.0837136Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:07.0837361Z ##[endgroup] 2025-12-04T09:25:07.0850259Z ##[group]Run set -x 2025-12-04T09:25:07.0850496Z set -x 2025-12-04T09:25:07.0850692Z set +e 2025-12-04T09:25:07.0850879Z  2025-12-04T09:25:07.0851068Z login() { 2025-12-04T09:25:07.0851434Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-12-04T09:25:07.0851814Z } 2025-12-04T09:25:07.0851984Z  2025-12-04T09:25:07.0852199Z retry () { 2025-12-04T09:25:07.0852415Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-12-04T09:25:07.0852661Z } 2025-12-04T09:25:07.0852832Z  2025-12-04T09:25:07.0853024Z retry login "${DOCKER_REGISTRY}" 2025-12-04T09:25:07.0853254Z  2025-12-04T09:25:07.0853621Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-12-04T09:25:07.0854109Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-12-04T09:25:07.0854385Z  2025-12-04T09:25:07.0854556Z set -e 2025-12-04T09:25:07.0854822Z # ignore output since only exit code is used for conditional 2025-12-04T09:25:07.0855181Z # only pull docker image if it's not available locally 2025-12-04T09:25:07.0855588Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-12-04T09:25:07.0855968Z  retry docker pull "${DOCKER_IMAGE}" 2025-12-04T09:25:07.0856218Z fi 2025-12-04T09:25:07.0860759Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:25:07.0861046Z env: 2025-12-04T09:25:07.0861232Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:25:07.0861866Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:07.0862493Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:25:07.0862759Z ##[endgroup] 2025-12-04T09:25:07.0886656Z + set +e 2025-12-04T09:25:07.0891638Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:25:07.0896269Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:25:07.0901148Z + aws ecr get-login-password --region us-east-1 2025-12-04T09:25:07.0906215Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:25:07.5191787Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:25:07.5192440Z Configure a credential helper to remove this warning. See 2025-12-04T09:25:07.5192937Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:25:07.5193771Z Login Succeeded 2025-12-04T09:25:07.5197814Z 2025-12-04T09:25:07.5226543Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:07.5227191Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-12-04T09:25:07.7524877Z + IMAGE_SIZE=29010.187264442444 2025-12-04T09:25:07.7525229Z Compressed size of image in MB: 29010.187264442444 2025-12-04T09:25:07.7688091Z + echo 'Compressed size of image in MB: 29010.187264442444' 2025-12-04T09:25:07.7688395Z + set -e 2025-12-04T09:25:07.7689354Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:07.7690314Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:07.7691240Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:25:07.9888365Z pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a: Pulling from pytorch/ci-image 2025-12-04T09:25:07.9891510Z 63e5bc7682b8: Pulling fs layer 2025-12-04T09:25:07.9891807Z 6dc15eca5138: Pulling fs layer 2025-12-04T09:25:07.9892009Z 459f4df18f07: Pulling fs layer 2025-12-04T09:25:07.9892186Z 821085416919: Pulling fs layer 2025-12-04T09:25:07.9892367Z 3738646b3d92: Pulling fs layer 2025-12-04T09:25:07.9892551Z aa9a10a37b39: Pulling fs layer 2025-12-04T09:25:07.9892773Z 55bac08f3e18: Pulling fs layer 2025-12-04T09:25:07.9892958Z acf6468f6aad: Pulling fs layer 2025-12-04T09:25:07.9893144Z c0c31a0e69b7: Pulling fs layer 2025-12-04T09:25:07.9893330Z 0f7d0dc70d2d: Pulling fs layer 2025-12-04T09:25:07.9893514Z 58c4d15d3bc1: Pulling fs layer 2025-12-04T09:25:07.9893708Z 2f1cc47b61e1: Pulling fs layer 2025-12-04T09:25:07.9893905Z d2472a74103d: Pulling fs layer 2025-12-04T09:25:07.9894094Z d1fba7293688: Pulling fs layer 2025-12-04T09:25:07.9894301Z 4f4fb700ef54: Pulling fs layer 2025-12-04T09:25:07.9894497Z 6a9b03ce41a7: Pulling fs layer 2025-12-04T09:25:07.9894683Z 3519ac15be79: Pulling fs layer 2025-12-04T09:25:07.9894882Z dabb51b819a0: Pulling fs layer 2025-12-04T09:25:07.9895077Z 8f36833a24d0: Pulling fs layer 2025-12-04T09:25:07.9895263Z ab53c5a853a5: Pulling fs layer 2025-12-04T09:25:07.9895465Z 025a0e5e6ac1: Pulling fs layer 2025-12-04T09:25:07.9895645Z ce3394c8f210: Pulling fs layer 2025-12-04T09:25:07.9895828Z a5c3888c3a0c: Pulling fs layer 2025-12-04T09:25:07.9896008Z ed902a3a4e3b: Pulling fs layer 2025-12-04T09:25:07.9896194Z b4e1efca22be: Pulling fs layer 2025-12-04T09:25:07.9896379Z 79fcfd297d9f: Pulling fs layer 2025-12-04T09:25:07.9896556Z 7bd89c134b49: Pulling fs layer 2025-12-04T09:25:07.9896752Z b7d40b4fd1b9: Pulling fs layer 2025-12-04T09:25:07.9896950Z dd1cdd87320d: Pulling fs layer 2025-12-04T09:25:07.9897187Z c21a6e1cd03a: Pulling fs layer 2025-12-04T09:25:07.9897397Z 0aa3ea0b2754: Pulling fs layer 2025-12-04T09:25:07.9897601Z 74e0bdbb05d3: Pulling fs layer 2025-12-04T09:25:07.9897794Z ba3aee7dcf03: Pulling fs layer 2025-12-04T09:25:07.9897990Z 8c8d59b8759d: Pulling fs layer 2025-12-04T09:25:07.9898181Z 3c960865867b: Pulling fs layer 2025-12-04T09:25:07.9898364Z 074db79e3832: Pulling fs layer 2025-12-04T09:25:07.9898557Z 226748f4ff23: Pulling fs layer 2025-12-04T09:25:07.9898791Z 75d817336f1d: Pulling fs layer 2025-12-04T09:25:07.9898966Z 850870a17b94: Pulling fs layer 2025-12-04T09:25:07.9899148Z e6225129924f: Pulling fs layer 2025-12-04T09:25:07.9899337Z 5cd6b9ec5c3c: Pulling fs layer 2025-12-04T09:25:07.9899521Z 613d2b1f3db8: Pulling fs layer 2025-12-04T09:25:07.9899695Z 388169fffe8a: Pulling fs layer 2025-12-04T09:25:07.9899875Z 6ca498e78b7b: Pulling fs layer 2025-12-04T09:25:07.9900055Z 952b753ec7be: Pulling fs layer 2025-12-04T09:25:07.9900232Z 5b6f6de4bdd1: Pulling fs layer 2025-12-04T09:25:07.9900438Z fa23d9952f3c: Pulling fs layer 2025-12-04T09:25:07.9900623Z c2aa9ea4d09a: Pulling fs layer 2025-12-04T09:25:07.9900805Z a9bbc9e426d3: Pulling fs layer 2025-12-04T09:25:07.9900996Z 82f2371f6dc2: Pulling fs layer 2025-12-04T09:25:07.9901181Z c87301921afa: Pulling fs layer 2025-12-04T09:25:07.9901356Z 384d566a822f: Pulling fs layer 2025-12-04T09:25:07.9901541Z 32ddcdd5dd69: Pulling fs layer 2025-12-04T09:25:07.9901728Z a116a32ceaa1: Pulling fs layer 2025-12-04T09:25:07.9901907Z f6fbbbe6067e: Pulling fs layer 2025-12-04T09:25:07.9902087Z 648018658875: Pulling fs layer 2025-12-04T09:25:07.9902295Z 5e2f7eac20ad: Pulling fs layer 2025-12-04T09:25:07.9902724Z 0d1243af7593: Pulling fs layer 2025-12-04T09:25:07.9902915Z 1f6d5b941ea1: Pulling fs layer 2025-12-04T09:25:07.9903095Z 4444f70d73ce: Pulling fs layer 2025-12-04T09:25:07.9903284Z bf45550ddbad: Pulling fs layer 2025-12-04T09:25:07.9903474Z 30fff4f4bad3: Pulling fs layer 2025-12-04T09:25:07.9903656Z cca4db1ba155: Pulling fs layer 2025-12-04T09:25:07.9903844Z f0ffa379f4eb: Pulling fs layer 2025-12-04T09:25:07.9904026Z df645e678f95: Pulling fs layer 2025-12-04T09:25:07.9904290Z 52ad6c17d103: Pulling fs layer 2025-12-04T09:25:07.9904472Z 34408374e32c: Pulling fs layer 2025-12-04T09:25:07.9904650Z 27803b661d9e: Pulling fs layer 2025-12-04T09:25:07.9904824Z 07dafc893dea: Pulling fs layer 2025-12-04T09:25:07.9905003Z 0aa3ea0b2754: Waiting 2025-12-04T09:25:07.9905173Z 74e0bdbb05d3: Waiting 2025-12-04T09:25:07.9905330Z ba3aee7dcf03: Waiting 2025-12-04T09:25:07.9905492Z 8c8d59b8759d: Waiting 2025-12-04T09:25:07.9905650Z 3c960865867b: Waiting 2025-12-04T09:25:07.9905798Z 074db79e3832: Waiting 2025-12-04T09:25:07.9905958Z 226748f4ff23: Waiting 2025-12-04T09:25:07.9906115Z 75d817336f1d: Waiting 2025-12-04T09:25:07.9906263Z 850870a17b94: Waiting 2025-12-04T09:25:07.9906420Z e6225129924f: Waiting 2025-12-04T09:25:07.9906575Z 5cd6b9ec5c3c: Waiting 2025-12-04T09:25:07.9906759Z 613d2b1f3db8: Waiting 2025-12-04T09:25:07.9906908Z 388169fffe8a: Waiting 2025-12-04T09:25:07.9907063Z 6ca498e78b7b: Waiting 2025-12-04T09:25:07.9907220Z 952b753ec7be: Waiting 2025-12-04T09:25:07.9907370Z 5b6f6de4bdd1: Waiting 2025-12-04T09:25:07.9907534Z fa23d9952f3c: Waiting 2025-12-04T09:25:07.9907691Z c2aa9ea4d09a: Waiting 2025-12-04T09:25:07.9907843Z a9bbc9e426d3: Waiting 2025-12-04T09:25:07.9908004Z 82f2371f6dc2: Waiting 2025-12-04T09:25:07.9908160Z 821085416919: Waiting 2025-12-04T09:25:07.9908306Z c87301921afa: Waiting 2025-12-04T09:25:07.9908462Z 3738646b3d92: Waiting 2025-12-04T09:25:07.9908616Z 384d566a822f: Waiting 2025-12-04T09:25:07.9908772Z aa9a10a37b39: Waiting 2025-12-04T09:25:07.9908924Z 32ddcdd5dd69: Waiting 2025-12-04T09:25:07.9909086Z acf6468f6aad: Waiting 2025-12-04T09:25:07.9909245Z a116a32ceaa1: Waiting 2025-12-04T09:25:07.9909395Z c0c31a0e69b7: Waiting 2025-12-04T09:25:07.9909552Z f6fbbbe6067e: Waiting 2025-12-04T09:25:07.9909705Z 648018658875: Waiting 2025-12-04T09:25:07.9909852Z 2f1cc47b61e1: Waiting 2025-12-04T09:25:07.9910009Z 58c4d15d3bc1: Waiting 2025-12-04T09:25:07.9910161Z d2472a74103d: Waiting 2025-12-04T09:25:07.9910307Z d1fba7293688: Waiting 2025-12-04T09:25:07.9910461Z 4f4fb700ef54: Waiting 2025-12-04T09:25:07.9910619Z 0f7d0dc70d2d: Waiting 2025-12-04T09:25:07.9910768Z 3519ac15be79: Waiting 2025-12-04T09:25:07.9910925Z 6a9b03ce41a7: Waiting 2025-12-04T09:25:07.9911084Z dabb51b819a0: Waiting 2025-12-04T09:25:07.9911234Z ce3394c8f210: Waiting 2025-12-04T09:25:07.9911390Z ab53c5a853a5: Waiting 2025-12-04T09:25:07.9911545Z a5c3888c3a0c: Waiting 2025-12-04T09:25:07.9911697Z b4e1efca22be: Waiting 2025-12-04T09:25:07.9911856Z ed902a3a4e3b: Waiting 2025-12-04T09:25:07.9912013Z 025a0e5e6ac1: Waiting 2025-12-04T09:25:07.9912169Z 79fcfd297d9f: Waiting 2025-12-04T09:25:07.9912328Z 7bd89c134b49: Waiting 2025-12-04T09:25:07.9912485Z dd1cdd87320d: Waiting 2025-12-04T09:25:07.9912644Z c21a6e1cd03a: Waiting 2025-12-04T09:25:07.9912814Z b7d40b4fd1b9: Waiting 2025-12-04T09:25:07.9912983Z 5e2f7eac20ad: Waiting 2025-12-04T09:25:07.9913152Z 0d1243af7593: Waiting 2025-12-04T09:25:07.9913310Z 8f36833a24d0: Waiting 2025-12-04T09:25:07.9913474Z 1f6d5b941ea1: Waiting 2025-12-04T09:25:07.9913643Z 27803b661d9e: Waiting 2025-12-04T09:25:07.9913805Z f0ffa379f4eb: Waiting 2025-12-04T09:25:07.9913973Z 34408374e32c: Waiting 2025-12-04T09:25:07.9914129Z bf45550ddbad: Waiting 2025-12-04T09:25:07.9914278Z 4444f70d73ce: Waiting 2025-12-04T09:25:07.9914437Z 07dafc893dea: Waiting 2025-12-04T09:25:07.9914593Z df645e678f95: Waiting 2025-12-04T09:25:07.9914744Z 30fff4f4bad3: Waiting 2025-12-04T09:25:07.9914902Z cca4db1ba155: Waiting 2025-12-04T09:25:08.0612228Z 6dc15eca5138: Verifying Checksum 2025-12-04T09:25:08.0612717Z 6dc15eca5138: Download complete 2025-12-04T09:25:08.1335169Z 821085416919: Download complete 2025-12-04T09:25:08.2140026Z 3738646b3d92: Verifying Checksum 2025-12-04T09:25:08.2140342Z 3738646b3d92: Download complete 2025-12-04T09:25:08.3172431Z aa9a10a37b39: Download complete 2025-12-04T09:25:08.3387972Z 63e5bc7682b8: Verifying Checksum 2025-12-04T09:25:08.3388449Z 63e5bc7682b8: Download complete 2025-12-04T09:25:08.4095505Z 55bac08f3e18: Verifying Checksum 2025-12-04T09:25:08.4095819Z 55bac08f3e18: Download complete 2025-12-04T09:25:08.4269271Z acf6468f6aad: Verifying Checksum 2025-12-04T09:25:08.4269579Z acf6468f6aad: Download complete 2025-12-04T09:25:08.5320952Z 0f7d0dc70d2d: Verifying Checksum 2025-12-04T09:25:08.5321278Z 0f7d0dc70d2d: Download complete 2025-12-04T09:25:08.6043771Z 58c4d15d3bc1: Verifying Checksum 2025-12-04T09:25:08.6044058Z 58c4d15d3bc1: Download complete 2025-12-04T09:25:08.6897114Z 2f1cc47b61e1: Download complete 2025-12-04T09:25:08.7637537Z d2472a74103d: Download complete 2025-12-04T09:25:09.5650012Z c0c31a0e69b7: Verifying Checksum 2025-12-04T09:25:09.5650548Z c0c31a0e69b7: Download complete 2025-12-04T09:25:09.5666741Z 63e5bc7682b8: Pull complete 2025-12-04T09:25:09.5739027Z 4f4fb700ef54: Download complete 2025-12-04T09:25:09.5783862Z 6dc15eca5138: Pull complete 2025-12-04T09:25:09.6515376Z 6a9b03ce41a7: Verifying Checksum 2025-12-04T09:25:09.6515794Z 6a9b03ce41a7: Download complete 2025-12-04T09:25:09.7240154Z 3519ac15be79: Verifying Checksum 2025-12-04T09:25:09.7240770Z 3519ac15be79: Download complete 2025-12-04T09:25:09.8114568Z dabb51b819a0: Verifying Checksum 2025-12-04T09:25:09.8115083Z dabb51b819a0: Download complete 2025-12-04T09:25:09.9088459Z 8f36833a24d0: Verifying Checksum 2025-12-04T09:25:09.9088973Z 8f36833a24d0: Download complete 2025-12-04T09:25:09.9865105Z ab53c5a853a5: Verifying Checksum 2025-12-04T09:25:09.9865651Z ab53c5a853a5: Download complete 2025-12-04T09:25:10.0824372Z 025a0e5e6ac1: Verifying Checksum 2025-12-04T09:25:10.0829522Z 025a0e5e6ac1: Download complete 2025-12-04T09:25:10.1777707Z ce3394c8f210: Verifying Checksum 2025-12-04T09:25:10.1778196Z ce3394c8f210: Download complete 2025-12-04T09:25:10.2596892Z a5c3888c3a0c: Verifying Checksum 2025-12-04T09:25:10.2597214Z a5c3888c3a0c: Download complete 2025-12-04T09:25:10.3435813Z ed902a3a4e3b: Download complete 2025-12-04T09:25:10.4306305Z b4e1efca22be: Verifying Checksum 2025-12-04T09:25:10.4306825Z b4e1efca22be: Download complete 2025-12-04T09:25:10.5073966Z 79fcfd297d9f: Verifying Checksum 2025-12-04T09:25:10.5074282Z 79fcfd297d9f: Download complete 2025-12-04T09:25:10.5851379Z 7bd89c134b49: Verifying Checksum 2025-12-04T09:25:10.5851680Z 7bd89c134b49: Download complete 2025-12-04T09:25:11.1806898Z 459f4df18f07: Verifying Checksum 2025-12-04T09:25:11.1807189Z 459f4df18f07: Download complete 2025-12-04T09:25:11.2789008Z dd1cdd87320d: Verifying Checksum 2025-12-04T09:25:11.2789308Z dd1cdd87320d: Download complete 2025-12-04T09:25:11.3634599Z c21a6e1cd03a: Verifying Checksum 2025-12-04T09:25:11.3634919Z c21a6e1cd03a: Download complete 2025-12-04T09:25:11.4389203Z 0aa3ea0b2754: Verifying Checksum 2025-12-04T09:25:11.4394479Z 0aa3ea0b2754: Download complete 2025-12-04T09:25:11.5232133Z 74e0bdbb05d3: Verifying Checksum 2025-12-04T09:25:11.5232453Z 74e0bdbb05d3: Download complete 2025-12-04T09:25:11.7929729Z ba3aee7dcf03: Verifying Checksum 2025-12-04T09:25:11.7930070Z ba3aee7dcf03: Download complete 2025-12-04T09:25:11.8717419Z 8c8d59b8759d: Verifying Checksum 2025-12-04T09:25:11.8717932Z 8c8d59b8759d: Download complete 2025-12-04T09:25:11.9554778Z 3c960865867b: Download complete 2025-12-04T09:25:12.0538339Z 074db79e3832: Verifying Checksum 2025-12-04T09:25:12.0538665Z 074db79e3832: Download complete 2025-12-04T09:25:12.1622042Z 226748f4ff23: Verifying Checksum 2025-12-04T09:25:12.1625978Z 226748f4ff23: Download complete 2025-12-04T09:25:12.2932407Z 75d817336f1d: Verifying Checksum 2025-12-04T09:25:12.2932728Z 75d817336f1d: Download complete 2025-12-04T09:25:15.2208279Z b7d40b4fd1b9: Verifying Checksum 2025-12-04T09:25:15.2208590Z b7d40b4fd1b9: Download complete 2025-12-04T09:25:15.2879892Z e6225129924f: Download complete 2025-12-04T09:25:15.3476595Z 5cd6b9ec5c3c: Verifying Checksum 2025-12-04T09:25:15.3476914Z 5cd6b9ec5c3c: Download complete 2025-12-04T09:25:15.4224733Z 613d2b1f3db8: Verifying Checksum 2025-12-04T09:25:15.4227873Z 613d2b1f3db8: Download complete 2025-12-04T09:25:18.6064720Z 388169fffe8a: Verifying Checksum 2025-12-04T09:25:18.6065114Z 388169fffe8a: Download complete 2025-12-04T09:25:22.9591179Z 459f4df18f07: Pull complete 2025-12-04T09:25:23.2814386Z 821085416919: Pull complete 2025-12-04T09:25:23.5319512Z 3738646b3d92: Pull complete 2025-12-04T09:25:23.8171336Z aa9a10a37b39: Pull complete 2025-12-04T09:25:24.1481121Z 55bac08f3e18: Pull complete 2025-12-04T09:25:24.4375674Z acf6468f6aad: Pull complete 2025-12-04T09:25:27.9449581Z c0c31a0e69b7: Pull complete 2025-12-04T09:25:28.2643591Z 0f7d0dc70d2d: Pull complete 2025-12-04T09:25:28.5113525Z 58c4d15d3bc1: Pull complete 2025-12-04T09:25:28.7956067Z 2f1cc47b61e1: Pull complete 2025-12-04T09:25:29.0449483Z d2472a74103d: Pull complete 2025-12-04T09:25:42.3380934Z d1fba7293688: Verifying Checksum 2025-12-04T09:25:42.3381234Z d1fba7293688: Download complete 2025-12-04T09:25:42.4230121Z 952b753ec7be: Download complete 2025-12-04T09:25:42.4969743Z 5b6f6de4bdd1: Verifying Checksum 2025-12-04T09:25:42.4975163Z 5b6f6de4bdd1: Download complete 2025-12-04T09:25:42.5868069Z fa23d9952f3c: Verifying Checksum 2025-12-04T09:25:42.5868534Z fa23d9952f3c: Download complete 2025-12-04T09:25:42.6774967Z c2aa9ea4d09a: Verifying Checksum 2025-12-04T09:25:42.6775296Z c2aa9ea4d09a: Download complete 2025-12-04T09:25:42.7619900Z a9bbc9e426d3: Verifying Checksum 2025-12-04T09:25:42.7620203Z a9bbc9e426d3: Download complete 2025-12-04T09:25:42.8360567Z 82f2371f6dc2: Verifying Checksum 2025-12-04T09:25:42.8360881Z 82f2371f6dc2: Download complete 2025-12-04T09:25:42.9222930Z c87301921afa: Verifying Checksum 2025-12-04T09:25:42.9227669Z c87301921afa: Download complete 2025-12-04T09:25:43.0465409Z 384d566a822f: Verifying Checksum 2025-12-04T09:25:43.0465758Z 384d566a822f: Download complete 2025-12-04T09:25:43.1775055Z 32ddcdd5dd69: Verifying Checksum 2025-12-04T09:25:43.1775393Z 32ddcdd5dd69: Download complete 2025-12-04T09:25:43.2698342Z a116a32ceaa1: Download complete 2025-12-04T09:25:43.3465279Z f6fbbbe6067e: Verifying Checksum 2025-12-04T09:25:43.3465590Z f6fbbbe6067e: Download complete 2025-12-04T09:25:43.5463084Z 5e2f7eac20ad: Verifying Checksum 2025-12-04T09:25:43.5465854Z 5e2f7eac20ad: Download complete 2025-12-04T09:25:43.6485721Z 0d1243af7593: Verifying Checksum 2025-12-04T09:25:43.6490512Z 0d1243af7593: Download complete 2025-12-04T09:25:43.7251131Z 1f6d5b941ea1: Verifying Checksum 2025-12-04T09:25:43.7252141Z 1f6d5b941ea1: Download complete 2025-12-04T09:25:43.8129100Z 4444f70d73ce: Verifying Checksum 2025-12-04T09:25:43.8134390Z 4444f70d73ce: Download complete 2025-12-04T09:25:43.8863715Z bf45550ddbad: Verifying Checksum 2025-12-04T09:25:43.8869453Z bf45550ddbad: Download complete 2025-12-04T09:25:46.3837962Z 30fff4f4bad3: Verifying Checksum 2025-12-04T09:25:46.3838389Z 30fff4f4bad3: Download complete 2025-12-04T09:25:46.4704138Z cca4db1ba155: Verifying Checksum 2025-12-04T09:25:46.4704459Z cca4db1ba155: Download complete 2025-12-04T09:25:46.5505096Z f0ffa379f4eb: Download complete 2025-12-04T09:25:46.6364772Z df645e678f95: Download complete 2025-12-04T09:25:46.7336558Z 52ad6c17d103: Verifying Checksum 2025-12-04T09:25:46.7336932Z 52ad6c17d103: Download complete 2025-12-04T09:25:46.8158622Z 34408374e32c: Verifying Checksum 2025-12-04T09:25:46.8158973Z 34408374e32c: Download complete 2025-12-04T09:25:46.8901469Z 27803b661d9e: Verifying Checksum 2025-12-04T09:25:46.8902442Z 27803b661d9e: Download complete 2025-12-04T09:25:47.5082189Z 07dafc893dea: Verifying Checksum 2025-12-04T09:25:47.5082498Z 07dafc893dea: Download complete 2025-12-04T09:26:24.9400404Z 6ca498e78b7b: Verifying Checksum 2025-12-04T09:26:24.9400746Z 6ca498e78b7b: Download complete 2025-12-04T09:27:02.5510287Z d1fba7293688: Pull complete 2025-12-04T09:27:02.8332383Z 4f4fb700ef54: Pull complete 2025-12-04T09:27:03.1758529Z 6a9b03ce41a7: Pull complete 2025-12-04T09:27:03.4319178Z 3519ac15be79: Pull complete 2025-12-04T09:27:03.4926426Z dabb51b819a0: Pull complete 2025-12-04T09:27:03.8089768Z 8f36833a24d0: Pull complete 2025-12-04T09:27:04.1801835Z ab53c5a853a5: Pull complete 2025-12-04T09:27:04.6344955Z 025a0e5e6ac1: Pull complete 2025-12-04T09:27:05.1258190Z ce3394c8f210: Pull complete 2025-12-04T09:27:05.4997866Z a5c3888c3a0c: Pull complete 2025-12-04T09:27:05.8332827Z ed902a3a4e3b: Pull complete 2025-12-04T09:27:06.2138454Z b4e1efca22be: Pull complete 2025-12-04T09:27:07.0667332Z 79fcfd297d9f: Pull complete 2025-12-04T09:27:07.6182194Z 7bd89c134b49: Pull complete 2025-12-04T09:27:19.7019999Z b7d40b4fd1b9: Pull complete 2025-12-04T09:27:20.0597349Z dd1cdd87320d: Pull complete 2025-12-04T09:27:20.4829243Z c21a6e1cd03a: Pull complete 2025-12-04T09:27:21.1558742Z 0aa3ea0b2754: Pull complete 2025-12-04T09:27:21.5740627Z 74e0bdbb05d3: Pull complete 2025-12-04T09:27:22.3440668Z ba3aee7dcf03: Pull complete 2025-12-04T09:27:22.7838048Z 8c8d59b8759d: Pull complete 2025-12-04T09:27:23.2399693Z 3c960865867b: Pull complete 2025-12-04T09:27:24.1559078Z 074db79e3832: Pull complete 2025-12-04T09:27:24.5291026Z 226748f4ff23: Pull complete 2025-12-04T09:27:24.8697422Z 75d817336f1d: Pull complete 2025-12-04T09:28:21.2634502Z 850870a17b94: Verifying Checksum 2025-12-04T09:28:21.2634976Z 850870a17b94: Download complete 2025-12-04T09:32:37.7073415Z 850870a17b94: Pull complete 2025-12-04T09:32:37.7359642Z e6225129924f: Pull complete 2025-12-04T09:32:37.7617715Z 5cd6b9ec5c3c: Pull complete 2025-12-04T09:32:37.8183697Z 613d2b1f3db8: Pull complete 2025-12-04T09:32:40.8703822Z 388169fffe8a: Pull complete 2025-12-04T09:35:20.9004624Z 6ca498e78b7b: Pull complete 2025-12-04T09:35:21.3940876Z 952b753ec7be: Pull complete 2025-12-04T09:35:21.8654809Z 5b6f6de4bdd1: Pull complete 2025-12-04T09:35:22.6584667Z fa23d9952f3c: Pull complete 2025-12-04T09:35:23.2256700Z c2aa9ea4d09a: Pull complete 2025-12-04T09:35:23.6183795Z a9bbc9e426d3: Pull complete 2025-12-04T09:35:24.4496811Z 82f2371f6dc2: Pull complete 2025-12-04T09:35:25.0681996Z c87301921afa: Pull complete 2025-12-04T09:35:25.4468657Z 384d566a822f: Pull complete 2025-12-04T09:35:26.2431276Z 32ddcdd5dd69: Pull complete 2025-12-04T09:35:27.0898446Z a116a32ceaa1: Pull complete 2025-12-04T09:35:27.6198462Z f6fbbbe6067e: Pull complete 2025-12-04T09:35:28.5997972Z 648018658875: Pull complete 2025-12-04T09:35:28.9312437Z 5e2f7eac20ad: Pull complete 2025-12-04T09:35:29.2339676Z 0d1243af7593: Pull complete 2025-12-04T09:35:29.2638262Z 1f6d5b941ea1: Pull complete 2025-12-04T09:35:29.3215486Z 4444f70d73ce: Pull complete 2025-12-04T09:35:29.3477256Z bf45550ddbad: Pull complete 2025-12-04T09:35:38.8670900Z 30fff4f4bad3: Pull complete 2025-12-04T09:35:39.3312875Z cca4db1ba155: Pull complete 2025-12-04T09:35:39.8387278Z f0ffa379f4eb: Pull complete 2025-12-04T09:35:40.3534647Z df645e678f95: Pull complete 2025-12-04T09:35:40.7950407Z 52ad6c17d103: Pull complete 2025-12-04T09:35:41.0369360Z 34408374e32c: Pull complete 2025-12-04T09:35:41.9064084Z 27803b661d9e: Pull complete 2025-12-04T09:35:44.3530185Z 07dafc893dea: Pull complete 2025-12-04T09:35:45.0213328Z Digest: sha256:b178ee928adfcab963ebbc6ea05ea1a0f0c605bb095e9076f010bf92d150869d 2025-12-04T09:35:45.1026942Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:35:45.1255762Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:35:45.1307861Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:35:45.1308513Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:35:45.1316860Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:35:45.1317139Z env: 2025-12-04T09:35:45.1317323Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:35:45.1317518Z ##[endgroup] 2025-12-04T09:35:45.1460289Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2025-12-04T09:35:45.1460594Z with: 2025-12-04T09:35:45.1460775Z driver-version: 580.82.07 2025-12-04T09:35:45.1460968Z env: 2025-12-04T09:35:45.1461142Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:35:45.1461517Z ##[endgroup] 2025-12-04T09:35:45.1583333Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:35:45.1583922Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:35:45.1588902Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:35:45.1589166Z env: 2025-12-04T09:35:45.1589340Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:35:45.1589530Z ##[endgroup] 2025-12-04T09:35:45.1751209Z ##[group]Run set -euo pipefail 2025-12-04T09:35:45.1751475Z set -euo pipefail 2025-12-04T09:35:45.1751710Z  2025-12-04T09:35:45.1751874Z has_gpu=false 2025-12-04T09:35:45.1752069Z devices="" 2025-12-04T09:35:45.1752246Z  2025-12-04T09:35:45.1752462Z if command -v nvidia-smi >/dev/null 2>&1; then 2025-12-04T09:35:45.1752778Z  if nvidia-smi -L >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:35:45.1753064Z  has_gpu=true 2025-12-04T09:35:45.1753290Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:35:45.1753511Z  fi 2025-12-04T09:35:45.1753684Z fi 2025-12-04T09:35:45.1753844Z  2025-12-04T09:35:45.1754003Z if [ "$has_gpu" = false ]; then 2025-12-04T09:35:45.1754276Z  if ls /dev/nvidia* >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:35:45.1754534Z  has_gpu=true 2025-12-04T09:35:45.1754747Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:35:45.1755015Z  fi 2025-12-04T09:35:45.1755172Z fi 2025-12-04T09:35:45.1755331Z  2025-12-04T09:35:45.1755552Z if [ "$has_gpu" = false ] && command -v lspci >/dev/null 2>&1; then 2025-12-04T09:35:45.1755893Z  if lspci | grep -i 'nvidia' >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:35:45.1756167Z  has_gpu=true 2025-12-04T09:35:45.1756376Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:35:45.1756594Z  fi 2025-12-04T09:35:45.1756753Z fi 2025-12-04T09:35:45.1756899Z  2025-12-04T09:35:45.1757128Z printf 'HAS_NVIDIA=%s\n' "$has_gpu" >> "$GITHUB_OUTPUT" 2025-12-04T09:35:45.1757494Z printf 'DETECTED_DEVICES<> "$GITHUB_OUTPUT" 2025-12-04T09:35:45.1762666Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:35:45.1762942Z env: 2025-12-04T09:35:45.1763118Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:35:45.1763317Z ##[endgroup] 2025-12-04T09:35:45.3218622Z ##[group]Run if [ "${HAS_NVIDIA}" = "true" ]; then 2025-12-04T09:35:45.3218936Z if [ "${HAS_NVIDIA}" = "true" ]; then 2025-12-04T09:35:45.3219207Z  echo "HAS_NVIDIA_GPU=true" >> "${GITHUB_ENV}" 2025-12-04T09:35:45.3219559Z  echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2025-12-04T09:35:45.3219871Z else 2025-12-04T09:35:45.3220076Z  echo "HAS_NVIDIA_GPU=false" >> "${GITHUB_ENV}" 2025-12-04T09:35:45.3220312Z fi 2025-12-04T09:35:45.3225537Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:35:45.3225789Z env: 2025-12-04T09:35:45.3225962Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:35:45.3226154Z HAS_NVIDIA: false 2025-12-04T09:35:45.3226323Z ##[endgroup] 2025-12-04T09:35:45.3304550Z Prepare all required actions 2025-12-04T09:35:45.3348068Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-12-04T09:35:45.3348339Z with: 2025-12-04T09:35:45.3348974Z github-token: *** 2025-12-04T09:35:45.3349157Z env: 2025-12-04T09:35:45.3349319Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:35:45.3349519Z HAS_NVIDIA_GPU: false 2025-12-04T09:35:45.3349709Z ##[endgroup] 2025-12-04T09:35:45.3457927Z ##[group]Run set -eux 2025-12-04T09:35:45.3458137Z set -eux 2025-12-04T09:35:45.3458490Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-12-04T09:35:45.3463915Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:35:45.3464163Z env: 2025-12-04T09:35:45.3464329Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:35:45.3464524Z HAS_NVIDIA_GPU: false 2025-12-04T09:35:45.3464877Z GITHUB_TOKEN: *** 2025-12-04T09:35:45.3465043Z ##[endgroup] 2025-12-04T09:35:45.3489449Z + python3 .github/scripts/get_workflow_job_id.py 19923066595 i-0c5ea43668efc70fe 2025-12-04T09:35:47.9477312Z Setting output job-id=57118563290 2025-12-04T09:35:47.9477957Z Setting output job-name=periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:35:47.9582540Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-12-04T09:35:47.9582985Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-12-04T09:35:47.9583544Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-12-04T09:35:47.9584314Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:35:47.9589237Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:35:47.9589488Z env: 2025-12-04T09:35:47.9589649Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:35:47.9589837Z HAS_NVIDIA_GPU: false 2025-12-04T09:35:47.9589999Z JOB_ID: 57118563290 2025-12-04T09:35:47.9590347Z JOB_NAME: periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:35:47.9590768Z WORKFLOW_NAME: inductor-periodic 2025-12-04T09:35:47.9590967Z WORKFLOW_RUN_ID: 19923066595 2025-12-04T09:35:47.9591156Z MONITOR_LOG_INTERVAL: 5 2025-12-04T09:35:47.9591340Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-12-04T09:35:47.9591519Z ##[endgroup] 2025-12-04T09:35:48.2072476Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T09:35:48.4728862Z Collecting psutil==5.9.8 2025-12-04T09:35:48.4867187Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-12-04T09:35:48.5432671Z Collecting dataclasses_json==0.6.7 2025-12-04T09:35:48.5465125Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-12-04T09:35:48.5703571Z Collecting nvidia-ml-py==11.525.84 2025-12-04T09:35:48.5737818Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-12-04T09:35:48.6020526Z Collecting typing-inspect<1,>=0.4.0 2025-12-04T09:35:48.6050245Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-12-04T09:35:48.6862975Z Collecting marshmallow<4.0.0,>=3.18.0 2025-12-04T09:35:48.6894465Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-12-04T09:35:48.7339320Z Collecting packaging>=17.0 2025-12-04T09:35:48.7373751Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-12-04T09:35:48.7788527Z Collecting typing-extensions>=3.7.4 2025-12-04T09:35:48.7820488Z Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2025-12-04T09:35:48.8008857Z Collecting mypy-extensions>=0.3.0 2025-12-04T09:35:48.8036992Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-12-04T09:35:48.8859265Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-12-04T09:35:49.1360081Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.15.0 typing-inspect-0.9.0 2025-12-04T09:35:49.2955144Z Prepare all required actions 2025-12-04T09:35:49.2955500Z Getting action download info 2025-12-04T09:35:49.4739976Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-12-04T09:35:49.7226376Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-12-04T09:35:50.0294649Z ##[group]Run ./.github/actions/download-build-artifacts 2025-12-04T09:35:50.0294980Z with: 2025-12-04T09:35:50.0295202Z name: linux-jammy-py3.10-gcc11-build 2025-12-04T09:35:50.0295467Z s3-bucket: gha-artifacts 2025-12-04T09:35:50.0295671Z env: 2025-12-04T09:35:50.0295854Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:35:50.0296066Z HAS_NVIDIA_GPU: false 2025-12-04T09:35:50.0296264Z ##[endgroup] 2025-12-04T09:35:50.0318340Z ##[group]Run seemethere/download-artifact-s3@v4 2025-12-04T09:35:50.0318599Z with: 2025-12-04T09:35:50.0318798Z name: linux-jammy-py3.10-gcc11-build 2025-12-04T09:35:50.0319090Z s3-bucket: gha-artifacts 2025-12-04T09:35:50.0319304Z region: us-east-1 2025-12-04T09:35:50.0319472Z env: 2025-12-04T09:35:50.0319641Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:35:50.0319843Z HAS_NVIDIA_GPU: false 2025-12-04T09:35:50.0320030Z ##[endgroup] 2025-12-04T09:35:50.4241907Z (node:48627) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-12-04T09:35:50.4242484Z 2025-12-04T09:35:50.4248533Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-12-04T09:35:50.4250393Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-12-04T09:35:50.4250811Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-12-04T09:35:50.7370143Z Found 1 objects with prefix pytorch/pytorch/19923066595/linux-jammy-py3.10-gcc11-build/ 2025-12-04T09:35:50.7375100Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-12-04T09:35:55.4533905Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-12-04T09:35:55.4539399Z Artifact download has finished successfully 2025-12-04T09:35:55.4739493Z ##[group]Run unzip -o artifacts.zip 2025-12-04T09:35:55.4739735Z unzip -o artifacts.zip 2025-12-04T09:35:55.4745333Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:35:55.4745608Z env: 2025-12-04T09:35:55.4745767Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:35:55.4745964Z HAS_NVIDIA_GPU: false 2025-12-04T09:35:55.4746151Z ##[endgroup] 2025-12-04T09:35:55.4817891Z Archive: artifacts.zip 2025-12-04T09:35:55.4821742Z creating: dist/ 2025-12-04T09:35:56.6063042Z inflating: dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl 2025-12-04T09:35:56.6063472Z creating: dist/vision/ 2025-12-04T09:35:56.6138201Z inflating: dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:35:56.6143393Z creating: dist/audio/ 2025-12-04T09:35:56.6163098Z inflating: dist/audio/torchaudio-2.10.0a0+e90a398-cp310-cp310-linux_x86_64.whl 2025-12-04T09:35:56.6163610Z creating: dist/ao/ 2025-12-04T09:35:56.6199950Z inflating: dist/ao/torchao-0.7.0+git51c87b6e-py3-none-any.whl 2025-12-04T09:35:56.6311331Z inflating: dist/.ninja_log 2025-12-04T09:35:56.6314313Z creating: build/custom_test_artifacts/ 2025-12-04T09:35:56.6314632Z creating: build/custom_test_artifacts/custom-op-build/ 2025-12-04T09:35:56.6314985Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-12-04T09:35:56.6315386Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:35:56.6315862Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:35:56.6316294Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/ 2025-12-04T09:35:56.6316726Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:35:56.6317571Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:35:56.6318025Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:35:56.6318532Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:35:56.6319162Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:35:56.6319637Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:35:56.6320112Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:35:56.6320704Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:35:56.6321271Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:35:56.6321838Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:35:56.6322481Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:35:56.6329549Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:35:56.6330361Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:35:56.6331004Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:35:56.6331944Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-12-04T09:35:56.6332557Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-12-04T09:35:56.6333166Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-12-04T09:35:56.6333815Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-12-04T09:35:56.6334417Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-12-04T09:35:56.6335007Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-12-04T09:35:56.6335481Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-12-04T09:35:56.6335986Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-12-04T09:35:56.6336469Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-12-04T09:35:56.6336935Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-12-04T09:35:56.6337393Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-12-04T09:35:56.6354018Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-12-04T09:35:56.6530600Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-12-04T09:35:56.6531338Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-12-04T09:35:56.6532333Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-12-04T09:35:56.6532941Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-12-04T09:35:56.6533759Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-12-04T09:35:56.6534235Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-12-04T09:35:56.6534770Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-12-04T09:35:56.6535593Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-12-04T09:35:56.6536090Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-12-04T09:35:56.6536572Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-12-04T09:35:56.6537137Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-12-04T09:35:56.6550099Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-12-04T09:35:56.6621632Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-12-04T09:35:56.6622227Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:35:56.6622700Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:35:56.6623136Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-12-04T09:35:56.6625222Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-12-04T09:35:56.6625731Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-12-04T09:35:56.6626237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-12-04T09:35:56.6627190Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-12-04T09:35:56.6627650Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-12-04T09:35:56.6781495Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-12-04T09:35:56.6837512Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-12-04T09:35:56.6838010Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-12-04T09:35:56.6838385Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-12-04T09:35:56.6838776Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:35:56.6839218Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:35:56.6839656Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/ 2025-12-04T09:35:56.6840092Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:35:56.6840645Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:35:56.6841112Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:35:56.6841647Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:35:56.6842259Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:35:56.6842727Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:35:56.6843234Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:35:56.6843664Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:35:56.6844200Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:35:56.6844746Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:35:56.6845253Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:35:56.6845784Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:35:56.6846397Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:35:56.6847399Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:35:56.6847866Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-12-04T09:35:56.6848311Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-12-04T09:35:56.6848810Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-12-04T09:35:56.6849528Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-12-04T09:35:56.6850058Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-12-04T09:35:56.6850499Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-12-04T09:35:56.6850968Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-12-04T09:35:56.6852088Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-12-04T09:35:56.6852555Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-12-04T09:35:56.6853006Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-12-04T09:35:56.6853501Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-12-04T09:35:56.6870305Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-12-04T09:35:56.6925837Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-12-04T09:35:56.6926668Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:35:56.6927334Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:35:56.6931822Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-12-04T09:35:56.6932437Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-12-04T09:35:56.6932991Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-12-04T09:35:56.6933392Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-12-04T09:35:56.6933778Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-12-04T09:35:56.6934145Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-12-04T09:35:56.6968122Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-12-04T09:35:56.6968538Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-12-04T09:35:56.6968894Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-12-04T09:35:56.6969310Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:35:56.6969778Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:35:56.6970204Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/ 2025-12-04T09:35:56.6970628Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:35:56.6971087Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:35:56.6971537Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:35:56.6976269Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:35:56.6976846Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:35:56.6977335Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:35:56.6978072Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:35:56.6978529Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:35:56.6979056Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:35:56.6979679Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:35:56.6980165Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:35:56.6980681Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:35:56.6981798Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:35:56.6982433Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:35:56.6982911Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-12-04T09:35:56.6983378Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-12-04T09:35:56.6983886Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-12-04T09:35:56.6984471Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-12-04T09:35:56.6985040Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-12-04T09:35:56.6985561Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-12-04T09:35:56.6986087Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-12-04T09:35:56.6986612Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-12-04T09:35:56.6987358Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-12-04T09:35:56.6987877Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-12-04T09:35:56.6988407Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-12-04T09:35:56.6988990Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-12-04T09:35:56.7099445Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-12-04T09:35:56.7104252Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-12-04T09:35:56.7108640Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-12-04T09:35:56.7114301Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-12-04T09:35:56.7114911Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-12-04T09:35:56.7115425Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-12-04T09:35:56.7115970Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-12-04T09:35:56.7116507Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-12-04T09:35:56.7117034Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-12-04T09:35:56.7117572Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-12-04T09:35:56.7118397Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-12-04T09:35:56.7118992Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-12-04T09:35:56.7169188Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-12-04T09:35:56.7170206Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:35:56.7170717Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:35:56.7171184Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-12-04T09:35:56.7171584Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-12-04T09:35:56.7171996Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-12-04T09:35:56.7172391Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-12-04T09:35:56.7172744Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-12-04T09:35:56.7173101Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-12-04T09:35:56.7261485Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-12-04T09:35:56.7301010Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-12-04T09:35:56.7303024Z creating: build/lib/ 2025-12-04T09:35:56.7371638Z inflating: build/lib/libprotobuf-lite.a 2025-12-04T09:35:56.7774423Z inflating: build/lib/libprotobuf.a 2025-12-04T09:35:56.8237980Z inflating: build/lib/libprotoc.a 2025-12-04T09:35:56.8247493Z inflating: build/lib/libpthreadpool.a 2025-12-04T09:35:56.8254601Z inflating: build/lib/libcpuinfo.a 2025-12-04T09:35:56.8261593Z inflating: build/lib/libcpuinfo_internals.a 2025-12-04T09:35:56.8264083Z inflating: build/lib/libclog.a 2025-12-04T09:35:56.8278840Z inflating: build/lib/libpytorch_qnnpack.a 2025-12-04T09:35:56.8280908Z inflating: build/lib/libnnpack_reference_layers.a 2025-12-04T09:35:56.8456034Z inflating: build/lib/libmicrokernels-prod.a 2025-12-04T09:35:56.8473073Z inflating: build/lib/libnnpack.a 2025-12-04T09:35:56.9271830Z inflating: build/lib/libmicrokernels-all.a 2025-12-04T09:35:56.9332098Z inflating: build/lib/libgtest.a 2025-12-04T09:35:56.9350281Z inflating: build/lib/libgmock.a 2025-12-04T09:35:56.9354623Z inflating: build/lib/libgtest_main.a 2025-12-04T09:35:56.9356902Z inflating: build/lib/libgmock_main.a 2025-12-04T09:35:56.9433923Z inflating: build/lib/libXNNPACK.a 2025-12-04T09:35:56.9503053Z inflating: build/lib/libbenchmark.a 2025-12-04T09:35:56.9506925Z inflating: build/lib/libbenchmark_main.a 2025-12-04T09:35:56.9507279Z inflating: build/lib/libjitprofiling.a 2025-12-04T09:35:56.9532211Z inflating: build/lib/libittnotify.a 2025-12-04T09:35:56.9571603Z inflating: build/lib/libasmjit.a 2025-12-04T09:35:57.0634594Z inflating: build/lib/libfbgemm.a 2025-12-04T09:35:57.0662766Z inflating: build/lib/libtensorpipe_uv.a 2025-12-04T09:35:57.1163783Z inflating: build/lib/libtensorpipe.a 2025-12-04T09:35:57.1276469Z inflating: build/lib/libgloo.a 2025-12-04T09:35:57.1317477Z inflating: build/lib/libonnx_proto.a 2025-12-04T09:35:57.1977684Z inflating: build/lib/libonnx.a 2025-12-04T09:35:58.1126861Z inflating: build/lib/libdnnl.a 2025-12-04T09:35:58.1144752Z inflating: build/lib/libfmt.a 2025-12-04T09:35:58.1406025Z inflating: build/lib/libkineto.a 2025-12-04T09:35:58.1510256Z inflating: build/lib/libc10.so 2025-12-04T09:35:58.1510654Z inflating: build/lib/libtorch_global_deps.so 2025-12-04T09:36:00.9102258Z inflating: build/lib/libtorch_cpu.so 2025-12-04T09:36:00.9107474Z inflating: build/lib/libtorch.so 2025-12-04T09:36:00.9172736Z inflating: build/lib/libtorchbind_test.so 2025-12-04T09:36:00.9193302Z inflating: build/lib/libjitbackend_test.so 2025-12-04T09:36:00.9216486Z inflating: build/lib/libbackend_with_compiler.so 2025-12-04T09:36:00.9241657Z inflating: build/lib/libaoti_custom_ops.so 2025-12-04T09:36:00.9246721Z inflating: build/lib/libshm.so 2025-12-04T09:36:01.1521478Z inflating: build/lib/libtorch_python.so 2025-12-04T09:36:01.1559468Z inflating: build/lib/libnnapi_backend.so 2025-12-04T09:36:01.1564821Z creating: build/bin/ 2025-12-04T09:36:01.1565087Z creating: build/bin/CMakeFiles/ 2025-12-04T09:36:01.1565353Z inflating: build/bin/cmake_install.cmake 2025-12-04T09:36:01.1565606Z inflating: build/bin/CTestTestfile.cmake 2025-12-04T09:36:01.2020524Z inflating: build/bin/protoc-3.13.0.0 2025-12-04T09:36:01.2488516Z inflating: build/bin/protoc 2025-12-04T09:36:01.2546189Z inflating: build/bin/c10_AllocatorConfig_test 2025-12-04T09:36:01.2602997Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-12-04T09:36:01.2663189Z inflating: build/bin/c10_DeviceGuard_test 2025-12-04T09:36:01.2721648Z inflating: build/bin/c10_Device_test 2025-12-04T09:36:01.2787188Z inflating: build/bin/c10_DispatchKeySet_test 2025-12-04T09:36:01.2848120Z inflating: build/bin/c10_Scalar_test 2025-12-04T09:36:01.2903574Z inflating: build/bin/c10_StreamGuard_test 2025-12-04T09:36:01.2968772Z inflating: build/bin/c10_SymInt_test 2025-12-04T09:36:01.3027664Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-12-04T09:36:01.3091885Z inflating: build/bin/c10_InlineStreamGuard_test 2025-12-04T09:36:01.3153267Z inflating: build/bin/c10_SizesAndStrides_test 2025-12-04T09:36:01.3230747Z inflating: build/bin/c10_cow_test 2025-12-04T09:36:01.3290311Z inflating: build/bin/c10_Bitset_test 2025-12-04T09:36:01.3346243Z inflating: build/bin/c10_ArrayRef_test 2025-12-04T09:36:01.3401677Z inflating: build/bin/c10_ConstexprCrc_test 2025-12-04T09:36:01.3458080Z inflating: build/bin/c10_DeadlockDetection_test 2025-12-04T09:36:01.3522331Z inflating: build/bin/c10_Enumerate_test 2025-12-04T09:36:01.3584541Z inflating: build/bin/c10_LeftRight_test 2025-12-04T09:36:01.3644335Z inflating: build/bin/c10_NetworkFlow_test 2025-12-04T09:36:01.3702008Z inflating: build/bin/c10_Half_test 2025-12-04T09:36:01.3761550Z inflating: build/bin/c10_IntrusiveList_test 2025-12-04T09:36:01.3818024Z inflating: build/bin/c10_Synchronized_test 2025-12-04T09:36:01.3880685Z inflating: build/bin/c10_ThreadLocal_test 2025-12-04T09:36:01.3938198Z inflating: build/bin/c10_Semaphore_test 2025-12-04T09:36:01.3999428Z inflating: build/bin/c10_bfloat16_test 2025-12-04T09:36:01.4057445Z inflating: build/bin/c10_accumulate_test 2025-12-04T09:36:01.4115753Z inflating: build/bin/c10_TypeIndex_test 2025-12-04T09:36:01.4177253Z inflating: build/bin/c10_exception_test 2025-12-04T09:36:01.4231526Z inflating: build/bin/c10_bit_cast_test 2025-12-04T09:36:01.4295199Z inflating: build/bin/c10_complex_math_test 2025-12-04T09:36:01.4351655Z inflating: build/bin/c10_error_test 2025-12-04T09:36:01.4409225Z inflating: build/bin/c10_flags_test 2025-12-04T09:36:01.4471365Z inflating: build/bin/c10_complex_test 2025-12-04T09:36:01.4530000Z inflating: build/bin/c10_irange_test 2025-12-04T09:36:01.4585706Z inflating: build/bin/c10_generic_math_test 2025-12-04T09:36:01.4642290Z inflating: build/bin/c10_nofatal_test 2025-12-04T09:36:01.4703694Z inflating: build/bin/c10_lazy_test 2025-12-04T09:36:01.4766443Z inflating: build/bin/c10_logging_test 2025-12-04T09:36:01.4935390Z inflating: build/bin/c10_intrusive_ptr_test 2025-12-04T09:36:01.5017060Z inflating: build/bin/c10_optional_test 2025-12-04T09:36:01.5078537Z inflating: build/bin/c10_registry_test 2025-12-04T09:36:01.5147620Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-12-04T09:36:01.5208953Z inflating: build/bin/c10_ssize_test 2025-12-04T09:36:01.5366243Z inflating: build/bin/c10_small_vector_test 2025-12-04T09:36:01.5428239Z inflating: build/bin/c10_string_util_test 2025-12-04T09:36:01.5483930Z inflating: build/bin/c10_string_view_test 2025-12-04T09:36:01.5540907Z inflating: build/bin/c10_tempfile_test 2025-12-04T09:36:01.5590978Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-12-04T09:36:01.5655156Z inflating: build/bin/c10_typeid_test 2025-12-04T09:36:01.6260274Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-12-04T09:36:01.6878651Z inflating: build/bin/vec_test_all_types_AVX512 2025-12-04T09:36:01.7503051Z inflating: build/bin/vec_test_all_types_AVX2 2025-12-04T09:36:01.7613765Z inflating: build/bin/test_aoti_abi_check 2025-12-04T09:36:01.7667997Z inflating: build/bin/test_vec_half_DEFAULT 2025-12-04T09:36:01.7723199Z inflating: build/bin/test_vec_half_AVX512 2025-12-04T09:36:01.7779653Z inflating: build/bin/test_vec_half_AVX2 2025-12-04T09:36:01.7838730Z inflating: build/bin/BackoffTest 2025-12-04T09:36:01.7899162Z inflating: build/bin/FileStoreTest 2025-12-04T09:36:01.7958301Z inflating: build/bin/static_runtime_bench 2025-12-04T09:36:01.8221955Z inflating: build/bin/static_runtime_test 2025-12-04T09:36:01.8302921Z inflating: build/bin/Dict_test 2025-12-04T09:36:01.8361886Z inflating: build/bin/Dimname_test 2025-12-04T09:36:01.8432172Z inflating: build/bin/MaybeOwned_test 2025-12-04T09:36:01.8499419Z inflating: build/bin/NamedTensor_test 2025-12-04T09:36:01.8560625Z inflating: build/bin/apply_utils_test 2025-12-04T09:36:01.8625645Z inflating: build/bin/atest 2025-12-04T09:36:01.8697693Z inflating: build/bin/basic 2025-12-04T09:36:01.8754631Z inflating: build/bin/broadcast_test 2025-12-04T09:36:01.8811204Z inflating: build/bin/cpu_allocator_test 2025-12-04T09:36:01.8875457Z inflating: build/bin/cpu_generator_test 2025-12-04T09:36:01.8934697Z inflating: build/bin/cpu_profiling_allocator_test 2025-12-04T09:36:01.9034713Z inflating: build/bin/cpu_rng_test 2025-12-04T09:36:01.9092105Z inflating: build/bin/dlconvertor_test 2025-12-04T09:36:01.9156173Z inflating: build/bin/extension_backend_test 2025-12-04T09:36:01.9217314Z inflating: build/bin/half_test 2025-12-04T09:36:01.9323312Z inflating: build/bin/ivalue_test 2025-12-04T09:36:01.9379221Z inflating: build/bin/lazy_tensor_test 2025-12-04T09:36:01.9438557Z inflating: build/bin/math_kernel_test 2025-12-04T09:36:01.9497741Z inflating: build/bin/memory_format_test 2025-12-04T09:36:01.9557463Z inflating: build/bin/memory_overlapping_test 2025-12-04T09:36:01.9617883Z inflating: build/bin/mobile_memory_cleanup 2025-12-04T09:36:01.9678448Z inflating: build/bin/native_test 2025-12-04T09:36:01.9734243Z inflating: build/bin/operator_name_test 2025-12-04T09:36:01.9790971Z inflating: build/bin/operators_test 2025-12-04T09:36:01.9852702Z inflating: build/bin/packedtensoraccessor_test 2025-12-04T09:36:01.9923705Z inflating: build/bin/pow_test 2025-12-04T09:36:01.9986960Z inflating: build/bin/quantized_test 2025-12-04T09:36:02.0045287Z inflating: build/bin/reduce_ops_test 2025-12-04T09:36:02.0103127Z inflating: build/bin/reportMemoryUsage_test 2025-12-04T09:36:02.0163860Z inflating: build/bin/scalar_tensor_test 2025-12-04T09:36:02.0229217Z inflating: build/bin/scalar_test 2025-12-04T09:36:02.0286887Z inflating: build/bin/StorageUtils_test 2025-12-04T09:36:02.0345401Z inflating: build/bin/stride_properties_test 2025-12-04T09:36:02.0428531Z inflating: build/bin/tensor_iterator_test 2025-12-04T09:36:02.0487839Z inflating: build/bin/test_parallel 2025-12-04T09:36:02.0543663Z inflating: build/bin/thread_init_test 2025-12-04T09:36:02.0609950Z inflating: build/bin/type_ptr_test 2025-12-04T09:36:02.0671904Z inflating: build/bin/type_test 2025-12-04T09:36:02.0728632Z inflating: build/bin/undefined_tensor_test 2025-12-04T09:36:02.0784300Z inflating: build/bin/verify_api_visibility 2025-12-04T09:36:02.0863768Z inflating: build/bin/legacy_vmap_test 2025-12-04T09:36:02.0920104Z inflating: build/bin/weakref_test 2025-12-04T09:36:02.0977522Z inflating: build/bin/wrapdim_test 2025-12-04T09:36:02.1034818Z inflating: build/bin/xla_tensor_test 2025-12-04T09:36:02.1100235Z inflating: build/bin/IListRef_test 2025-12-04T09:36:02.1212733Z inflating: build/bin/List_test 2025-12-04T09:36:02.1284917Z inflating: build/bin/KernelFunction_test 2025-12-04T09:36:02.1410897Z inflating: build/bin/kernel_function_legacy_test 2025-12-04T09:36:02.1511467Z inflating: build/bin/kernel_function_test 2025-12-04T09:36:02.1642491Z inflating: build/bin/kernel_lambda_legacy_test 2025-12-04T09:36:02.1750799Z inflating: build/bin/kernel_lambda_test 2025-12-04T09:36:02.1811273Z inflating: build/bin/kernel_stackbased_test 2025-12-04T09:36:02.1913730Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-12-04T09:36:02.1972134Z inflating: build/bin/CppSignature_test 2025-12-04T09:36:02.2031483Z inflating: build/bin/backend_fallback_test 2025-12-04T09:36:02.2086079Z inflating: build/bin/op_allowlist_test 2025-12-04T09:36:02.2396219Z inflating: build/bin/op_registration_test 2025-12-04T09:36:02.2468111Z inflating: build/bin/inline_container_test 2025-12-04T09:36:02.2811617Z inflating: build/bin/test_lazy 2025-12-04T09:36:02.2870503Z inflating: build/bin/TCPStoreTest 2025-12-04T09:36:02.2928730Z inflating: build/bin/HashStoreTest 2025-12-04T09:36:02.4022280Z inflating: build/bin/test_jit 2025-12-04T09:36:02.4093440Z inflating: build/bin/ProcessGroupGlooTest 2025-12-04T09:36:02.4156703Z inflating: build/bin/test_aoti_inference 2025-12-04T09:36:02.4157026Z inflating: build/bin/example_allreduce 2025-12-04T09:36:02.4217783Z inflating: build/bin/test_dist_autograd 2025-12-04T09:36:02.4294327Z inflating: build/bin/test_cpp_rpc 2025-12-04T09:36:02.4294656Z inflating: build/bin/parallel_benchmark 2025-12-04T09:36:02.5470737Z inflating: build/bin/test_api 2025-12-04T09:36:02.5471492Z inflating: build/bin/torch_shm_manager 2025-12-04T09:36:02.5471754Z creating: .additional_ci_files/ 2025-12-04T09:36:02.5535267Z inflating: .additional_ci_files/test-times.json 2025-12-04T09:36:02.5768644Z inflating: .additional_ci_files/test-class-times.json 2025-12-04T09:36:02.5793712Z ##[group]Run rm artifacts.zip 2025-12-04T09:36:02.5793948Z rm artifacts.zip 2025-12-04T09:36:02.5799204Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:36:02.5799470Z env: 2025-12-04T09:36:02.5799633Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:02.5799831Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:02.5800019Z ##[endgroup] 2025-12-04T09:36:02.6174047Z ##[group]Run df -H 2025-12-04T09:36:02.6174235Z df -H 2025-12-04T09:36:02.6179525Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:36:02.6179818Z env: 2025-12-04T09:36:02.6179985Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:02.6180190Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:02.6180367Z ##[endgroup] 2025-12-04T09:36:02.6228607Z Filesystem Size Used Avail Use% Mounted on 2025-12-04T09:36:02.6229123Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-12-04T09:36:02.6229468Z tmpfs 67G 0 67G 0% /dev/shm 2025-12-04T09:36:02.6230653Z tmpfs 27G 791k 27G 1% /run 2025-12-04T09:36:02.6231011Z /dev/nvme0n1p1 215G 72G 144G 34% / 2025-12-04T09:36:02.6231264Z tmpfs 67G 13k 67G 1% /tmp 2025-12-04T09:36:02.6231517Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-12-04T09:36:02.6256848Z Prepare all required actions 2025-12-04T09:36:02.6261143Z Getting action download info 2025-12-04T09:36:02.8094130Z ##[group]Run ./.github/actions/download-td-artifacts 2025-12-04T09:36:02.8094444Z with: 2025-12-04T09:36:02.8094628Z env: 2025-12-04T09:36:02.8094809Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:02.8095034Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:02.8095242Z ##[endgroup] 2025-12-04T09:36:02.8116682Z ##[group]Run seemethere/download-artifact-s3@v4 2025-12-04T09:36:02.8116942Z with: 2025-12-04T09:36:02.8117104Z name: td_results 2025-12-04T09:36:02.8117295Z s3-bucket: gha-artifacts 2025-12-04T09:36:02.8117499Z region: us-east-1 2025-12-04T09:36:02.8117783Z env: 2025-12-04T09:36:02.8117947Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:02.8118143Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:02.8118331Z ##[endgroup] 2025-12-04T09:36:03.2745173Z (node:48648) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-12-04T09:36:03.2745884Z 2025-12-04T09:36:03.2746117Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-12-04T09:36:03.2746642Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-12-04T09:36:03.2747293Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-12-04T09:36:03.3592427Z Found 0 objects with prefix pytorch/pytorch/19923066595/td_results/ 2025-12-04T09:36:03.3597613Z Artifact download has finished successfully 2025-12-04T09:36:03.3800759Z ##[group]Run mkdir -p .additional_ci_files 2025-12-04T09:36:03.3801062Z mkdir -p .additional_ci_files 2025-12-04T09:36:03.3801371Z mv td_results.json .additional_ci_files/td_results.json || true 2025-12-04T09:36:03.3806859Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:36:03.3807107Z env: 2025-12-04T09:36:03.3807275Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:03.3807458Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:03.3807637Z ##[endgroup] 2025-12-04T09:36:03.3858955Z mv: cannot stat 'td_results.json': No such file or directory 2025-12-04T09:36:03.3897811Z ##[group]Run .github/scripts/parse_ref.py 2025-12-04T09:36:03.3898114Z .github/scripts/parse_ref.py 2025-12-04T09:36:03.3902346Z shell: /usr/bin/bash -e {0} 2025-12-04T09:36:03.3902549Z env: 2025-12-04T09:36:03.3902716Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:03.3902918Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:03.3903096Z ##[endgroup] 2025-12-04T09:36:03.4090518Z Setting output branch=main 2025-12-04T09:36:03.4184062Z Prepare all required actions 2025-12-04T09:36:03.4184381Z Getting action download info 2025-12-04T09:36:03.5664293Z ##[group]Run ./.github/actions/filter-test-configs 2025-12-04T09:36:03.5664598Z with: 2025-12-04T09:36:03.5665019Z github-token: *** 2025-12-04T09:36:03.5671717Z test-matrix: {"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}]} 2025-12-04T09:36:03.5678974Z job-name: periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:36:03.5679472Z env: 2025-12-04T09:36:03.5679669Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:03.5680023Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:03.5680256Z ##[endgroup] 2025-12-04T09:36:03.5708898Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T09:36:03.5709154Z with: 2025-12-04T09:36:03.5709350Z shell: bash 2025-12-04T09:36:03.5709547Z timeout_minutes: 10 2025-12-04T09:36:03.5709747Z max_attempts: 5 2025-12-04T09:36:03.5709950Z retry_wait_seconds: 30 2025-12-04T09:36:03.5710557Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-12-04T09:36:03.5711172Z polling_interval_seconds: 1 2025-12-04T09:36:03.5711405Z warning_on_retry: true 2025-12-04T09:36:03.5711628Z continue_on_error: false 2025-12-04T09:36:03.5711853Z env: 2025-12-04T09:36:03.5712037Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:03.5712268Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:03.5712650Z GITHUB_TOKEN: *** 2025-12-04T09:36:03.5712859Z ##[endgroup] 2025-12-04T09:36:03.6631374Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-12-04T09:36:03.8585343Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T09:36:03.9695403Z Collecting requests==2.27.1 2025-12-04T09:36:03.9837703Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-12-04T09:36:04.1213091Z Collecting pyyaml==6.0.2 2025-12-04T09:36:04.1244709Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-12-04T09:36:04.4246807Z Collecting charset-normalizer~=2.0.0 2025-12-04T09:36:04.4282021Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-12-04T09:36:04.4339435Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-12-04T09:36:04.4705671Z Collecting certifi>=2017.4.17 2025-12-04T09:36:04.4737728Z Downloading certifi-2025.11.12-py3-none-any.whl (159 kB) 2025-12-04T09:36:04.4812127Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-12-04T09:36:04.5486820Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-12-04T09:36:04.6636394Z Successfully installed certifi-2025.11.12 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-12-04T09:36:05.6394280Z Command completed after 1 attempt(s). 2025-12-04T09:36:05.6460912Z ##[group]Run set -x 2025-12-04T09:36:05.6461123Z set -x 2025-12-04T09:36:05.6461320Z  2025-12-04T09:36:05.6461586Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-12-04T09:36:05.6461885Z # in runner workspace 2025-12-04T09:36:05.6462149Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-12-04T09:36:05.6467454Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:36:05.6467692Z env: 2025-12-04T09:36:05.6467871Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:05.6468066Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:05.6468277Z ##[endgroup] 2025-12-04T09:36:05.6491625Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-12-04T09:36:05.6809802Z Setting output branch=main 2025-12-04T09:36:05.6884170Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-12-04T09:36:05.6884499Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-12-04T09:36:05.6884738Z echo "Job name: ${JOB_NAME}" 2025-12-04T09:36:05.6884954Z  2025-12-04T09:36:05.6885228Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-12-04T09:36:05.6885648Z # in runner workspace 2025-12-04T09:36:05.6885987Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-12-04T09:36:05.6886334Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-12-04T09:36:05.6886580Z  --job-name "${JOB_NAME}" \ 2025-12-04T09:36:05.6893377Z  --test-matrix "{"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}]}" \ 2025-12-04T09:36:05.6900496Z  --selected-test-configs "" \ 2025-12-04T09:36:05.6900779Z  --pr-number "${PR_NUMBER}" \ 2025-12-04T09:36:05.6901047Z  --tag "${TAG}" \ 2025-12-04T09:36:05.6901281Z  --event-name "${EVENT_NAME}" \ 2025-12-04T09:36:05.6901540Z  --schedule "${SCHEDULE}" \ 2025-12-04T09:36:05.6901787Z  --branch "${HEAD_BRANCH}" 2025-12-04T09:36:05.6907097Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:36:05.6907412Z env: 2025-12-04T09:36:05.6907626Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:05.6907915Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:05.6908561Z GITHUB_TOKEN: *** 2025-12-04T09:36:05.6909040Z JOB_NAME: periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:36:05.6909539Z PR_NUMBER: 2025-12-04T09:36:05.6909734Z TAG: 2025-12-04T09:36:05.6909926Z EVENT_NAME: schedule 2025-12-04T09:36:05.6910159Z SCHEDULE: 45 0,4,8,12,16,20 * * 1-5 2025-12-04T09:36:05.6910407Z HEAD_BRANCH: main 2025-12-04T09:36:05.6910624Z ##[endgroup] 2025-12-04T09:36:05.6934114Z Workflow: inductor-periodic 2025-12-04T09:36:05.6934634Z Job name: periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:36:05.8943827Z Setting output keep-going=True 2025-12-04T09:36:05.8944294Z Setting output ci-verbose-test-logs=False 2025-12-04T09:36:05.8944685Z Setting output ci-test-showlocals=False 2025-12-04T09:36:05.8944924Z Setting output ci-no-test-timeout=False 2025-12-04T09:36:05.8945179Z Setting output ci-no-td=False 2025-12-04T09:36:05.8945410Z Setting output ci-td-distributed=False 2025-12-04T09:36:05.8945650Z Setting output is-unstable=False 2025-12-04T09:36:05.8945870Z Setting output reenabled-issues= 2025-12-04T09:36:05.8953025Z Setting output test-matrix={"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}]} 2025-12-04T09:36:05.8960005Z Setting output is-test-matrix-empty=False 2025-12-04T09:36:05.9103162Z ##[group]Run echo "Filtered matrix:" 2025-12-04T09:36:05.9103498Z echo "Filtered matrix:" 2025-12-04T09:36:05.9110451Z echo "{"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}]}" 2025-12-04T09:36:05.9117699Z  2025-12-04T09:36:05.9117885Z echo 2025-12-04T09:36:05.9118130Z echo "Is the current job unstable? False" 2025-12-04T09:36:05.9118411Z  2025-12-04T09:36:05.9118589Z echo 2025-12-04T09:36:05.9118818Z echo "Is keep-going label set? True" 2025-12-04T09:36:05.9119089Z  2025-12-04T09:36:05.9119257Z echo 2025-12-04T09:36:05.9119465Z echo "Reenabled issues? " 2025-12-04T09:36:05.9125258Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:36:05.9125571Z env: 2025-12-04T09:36:05.9125783Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:05.9126022Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:05.9126242Z ##[endgroup] 2025-12-04T09:36:05.9152950Z Filtered matrix: 2025-12-04T09:36:05.9159699Z {include: [{config: cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_avx2_huggingface, shard: 1, num_shards: 1, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_torchbench, shard: 1, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_torchbench, shard: 2, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_timm, shard: 1, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_timm, shard: 2, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_amp_freezing_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: cpu_inductor_amp_freezing_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_amp_freezing_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_amp_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_amp_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_freezing_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_freezing_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_freezing_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_amp_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_amp_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_aot_inductor_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_aot_inductor_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_aot_inductor_amp_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_aot_inductor_amp_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}]} 2025-12-04T09:36:05.9166616Z 2025-12-04T09:36:05.9166727Z Is the current job unstable? False 2025-12-04T09:36:05.9166898Z 2025-12-04T09:36:05.9166998Z Is keep-going label set? True 2025-12-04T09:36:05.9167160Z 2025-12-04T09:36:05.9167245Z Reenabled issues? 2025-12-04T09:36:05.9207796Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-12-04T09:36:05.9208175Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-12-04T09:36:05.9212569Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:36:05.9212826Z env: 2025-12-04T09:36:05.9212999Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:05.9213200Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:05.9213380Z JOB_TIMEOUT: 240 2025-12-04T09:36:05.9213555Z ##[endgroup] 2025-12-04T09:36:05.9324508Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:36:05.9324901Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:36:05.9325245Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:36:05.9329767Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:36:05.9330042Z env: 2025-12-04T09:36:05.9330220Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:05.9330427Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:05.9330615Z ##[endgroup] 2025-12-04T09:36:05.9434635Z ##[group]Run set -x 2025-12-04T09:36:05.9434895Z set -x 2025-12-04T09:36:05.9435068Z  2025-12-04T09:36:05.9435265Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-12-04T09:36:05.9435541Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-12-04T09:36:05.9435823Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-12-04T09:36:05.9436080Z  TEST_COMMAND=.ci/onnx/test.sh 2025-12-04T09:36:05.9436293Z else 2025-12-04T09:36:05.9436493Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-12-04T09:36:05.9436717Z fi 2025-12-04T09:36:05.9436984Z  2025-12-04T09:36:05.9437193Z # Leaving 1GB for the runner and other things 2025-12-04T09:36:05.9437606Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-12-04T09:36:05.9438206Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-12-04T09:36:05.9438689Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-12-04T09:36:05.9439065Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-12-04T09:36:05.9439359Z  2025-12-04T09:36:05.9439566Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-12-04T09:36:05.9439823Z  SHM_OPTS= 2025-12-04T09:36:05.9440022Z  JENKINS_USER= 2025-12-04T09:36:05.9440281Z  # ensure that docker container cleanly exits in 12 hours 2025-12-04T09:36:05.9440931Z  # if for some reason cleanup action doesn't stop container 2025-12-04T09:36:05.9441226Z  # when job is cancelled 2025-12-04T09:36:05.9441462Z  DOCKER_SHELL_CMD="sleep 12h" 2025-12-04T09:36:05.9441699Z  USED_IMAGE="${DOCKER_IMAGE_S390X}" 2025-12-04T09:36:05.9441923Z else 2025-12-04T09:36:05.9442123Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-12-04T09:36:05.9442363Z  JENKINS_USER="--user jenkins" 2025-12-04T09:36:05.9442595Z  DOCKER_SHELL_CMD= 2025-12-04T09:36:05.9442814Z  USED_IMAGE="${DOCKER_IMAGE}" 2025-12-04T09:36:05.9443018Z fi 2025-12-04T09:36:05.9443181Z  2025-12-04T09:36:05.9443428Z # detached container should get cleaned up by teardown_ec2_linux 2025-12-04T09:36:05.9443793Z # TODO: Stop building test binaries as part of the build phase 2025-12-04T09:36:05.9444190Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-12-04T09:36:05.9444551Z # shellcheck disable=SC2086,SC2090 2025-12-04T09:36:05.9444797Z container_name=$(docker run \ 2025-12-04T09:36:05.9445018Z  ${GPU_FLAG:-} \ 2025-12-04T09:36:05.9445248Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-12-04T09:36:05.9445501Z  -e BUILD_ENVIRONMENT \ 2025-12-04T09:36:05.9445723Z  -e PR_NUMBER \ 2025-12-04T09:36:05.9445927Z  -e GITHUB_ACTIONS \ 2025-12-04T09:36:05.9446145Z  -e GITHUB_REPOSITORY \ 2025-12-04T09:36:05.9446364Z  -e GITHUB_WORKFLOW \ 2025-12-04T09:36:05.9446568Z  -e GITHUB_JOB \ 2025-12-04T09:36:05.9446767Z  -e GITHUB_RUN_ID \ 2025-12-04T09:36:05.9446977Z  -e GITHUB_RUN_NUMBER \ 2025-12-04T09:36:05.9447488Z  -e GITHUB_RUN_ATTEMPT \ 2025-12-04T09:36:05.9447719Z  -e JOB_ID \ 2025-12-04T09:36:05.9447921Z  -e JOB_NAME \ 2025-12-04T09:36:05.9448124Z  -e BASE_SHA \ 2025-12-04T09:36:05.9448316Z  -e BRANCH \ 2025-12-04T09:36:05.9448508Z  -e SHA1 \ 2025-12-04T09:36:05.9448704Z  -e AWS_DEFAULT_REGION \ 2025-12-04T09:36:05.9448915Z  -e IN_WHEEL_TEST \ 2025-12-04T09:36:05.9449122Z  -e SHARD_NUMBER \ 2025-12-04T09:36:05.9449327Z  -e TEST_CONFIG \ 2025-12-04T09:36:05.9449524Z  -e NUM_TEST_SHARDS \ 2025-12-04T09:36:05.9449871Z  -e REENABLED_ISSUES \ 2025-12-04T09:36:05.9450104Z  -e CONTINUE_THROUGH_ERROR \ 2025-12-04T09:36:05.9450328Z  -e VERBOSE_TEST_LOGS \ 2025-12-04T09:36:05.9450545Z  -e TEST_SHOWLOCALS \ 2025-12-04T09:36:05.9450768Z  -e NO_TEST_TIMEOUT \ 2025-12-04T09:36:05.9450963Z  -e NO_TD \ 2025-12-04T09:36:05.9451146Z  -e TD_DISTRIBUTED \ 2025-12-04T09:36:05.9451353Z  -e PR_LABELS \ 2025-12-04T09:36:05.9451566Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-12-04T09:36:05.9451794Z  -e SCCACHE_BUCKET \ 2025-12-04T09:36:05.9452075Z  -e SCCACHE_REGION \ 2025-12-04T09:36:05.9452271Z  -e XLA_CUDA \ 2025-12-04T09:36:05.9452471Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-12-04T09:36:05.9452717Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-12-04T09:36:05.9452971Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-12-04T09:36:05.9453226Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-12-04T09:36:05.9453456Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-12-04T09:36:05.9453688Z  -e VLLM_TEST_HUGGING_FACE_TOKEN \ 2025-12-04T09:36:05.9453926Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-12-04T09:36:05.9454144Z  -e DASHBOARD_TAG \ 2025-12-04T09:36:05.9454351Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-12-04T09:36:05.9454602Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-12-04T09:36:05.9454876Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-12-04T09:36:05.9455162Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-12-04T09:36:05.9455437Z  --security-opt seccomp=unconfined \ 2025-12-04T09:36:05.9455672Z  --cap-add=SYS_PTRACE \ 2025-12-04T09:36:05.9455878Z  --ipc=host \ 2025-12-04T09:36:05.9456069Z  ${SHM_OPTS} \ 2025-12-04T09:36:05.9456256Z  --tty \ 2025-12-04T09:36:05.9456424Z  --detach \ 2025-12-04T09:36:05.9456623Z  --name="${container_name}" \ 2025-12-04T09:36:05.9456844Z  ${JENKINS_USER} \ 2025-12-04T09:36:05.9457085Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-12-04T09:36:05.9457360Z  -w /var/lib/jenkins/workspace \ 2025-12-04T09:36:05.9457588Z  "${USED_IMAGE}" \ 2025-12-04T09:36:05.9457789Z  ${DOCKER_SHELL_CMD} 2025-12-04T09:36:05.9457978Z ) 2025-12-04T09:36:05.9458220Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-12-04T09:36:05.9458498Z  2025-12-04T09:36:05.9458693Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-12-04T09:36:05.9459141Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-12-04T09:36:05.9459495Z fi 2025-12-04T09:36:05.9459686Z  2025-12-04T09:36:05.9460029Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-12-04T09:36:05.9464875Z shell: /usr/bin/bash -e {0} 2025-12-04T09:36:05.9465090Z env: 2025-12-04T09:36:05.9465268Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:36:05.9465488Z HAS_NVIDIA_GPU: false 2025-12-04T09:36:05.9465742Z BUILD_ENVIRONMENT: linux-jammy-py3.10-gcc11-build 2025-12-04T09:36:05.9465998Z PR_NUMBER: 2025-12-04T09:36:05.9466199Z GITHUB_REPOSITORY: pytorch/pytorch 2025-12-04T09:36:05.9466447Z GITHUB_WORKFLOW: inductor-periodic 2025-12-04T09:36:05.9466663Z GITHUB_JOB: test 2025-12-04T09:36:05.9466854Z GITHUB_RUN_ID: 19923066595 2025-12-04T09:36:05.9467077Z GITHUB_RUN_NUMBER: 67027 2025-12-04T09:36:05.9467274Z GITHUB_RUN_ATTEMPT: 1 2025-12-04T09:36:05.9467456Z JOB_ID: 57118563290 2025-12-04T09:36:05.9467837Z JOB_NAME: periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:36:05.9468236Z BRANCH: main 2025-12-04T09:36:05.9468444Z SHA1: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:36:05.9468827Z BASE_SHA: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:36:05.9469109Z TEST_CONFIG: dynamic_cpu_inductor_huggingface 2025-12-04T09:36:05.9469341Z SHARD_NUMBER: 1 2025-12-04T09:36:05.9469521Z NUM_TEST_SHARDS: 1 2025-12-04T09:36:05.9469699Z EXTRA_FLAGS: 2025-12-04T09:36:05.9469873Z OP_BENCHMARK_TESTS: 2025-12-04T09:36:05.9470055Z REENABLED_ISSUES: 2025-12-04T09:36:05.9470250Z CONTINUE_THROUGH_ERROR: True 2025-12-04T09:36:05.9470461Z VERBOSE_TEST_LOGS: False 2025-12-04T09:36:05.9470656Z TEST_SHOWLOCALS: False 2025-12-04T09:36:05.9470854Z NO_TEST_TIMEOUT: False 2025-12-04T09:36:05.9471108Z NO_TD: False 2025-12-04T09:36:05.9471278Z TD_DISTRIBUTED: False 2025-12-04T09:36:05.9471517Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-12-04T09:36:05.9471781Z SCCACHE_REGION: us-east-1 2025-12-04T09:36:05.9471975Z SHM_SIZE: 1g 2025-12-04T09:36:05.9472537Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:36:05.9473491Z DOCKER_IMAGE_S390X: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:36:05.9474073Z XLA_CUDA: 2025-12-04T09:36:05.9474372Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:36:05.9474722Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-12-04T09:36:05.9474965Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-12-04T09:36:05.9475211Z DASHBOARD_TAG: 2025-12-04T09:36:05.9475593Z VLLM_TEST_HUGGING_FACE_TOKEN: *** 2025-12-04T09:36:05.9475894Z HUGGING_FACE_HUB_TOKEN: *** 2025-12-04T09:36:05.9476194Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-12-04T09:36:05.9476568Z ARTIFACTS_FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290 2025-12-04T09:36:05.9476929Z ##[endgroup] 2025-12-04T09:36:05.9498120Z + [[ dynamic_cpu_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2025-12-04T09:36:05.9498513Z + [[ linux-jammy-py3.10-gcc11-build == *onnx* ]] 2025-12-04T09:36:05.9498809Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-12-04T09:36:05.9505654Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-12-04T09:36:06.0501505Z + TOTAL_AVAILABLE_MEMORY_IN_GB='122.780 ' 2025-12-04T09:36:06.0504287Z + TOTAL_MEMORY_WITH_SWAP=125 2025-12-04T09:36:06.0504581Z + [[ linux-jammy-py3.10-gcc11-build == *\s\3\9\0\x* ]] 2025-12-04T09:36:06.0505216Z + SHM_OPTS=--shm-size=1g 2025-12-04T09:36:06.0505495Z + JENKINS_USER='--user jenkins' 2025-12-04T09:36:06.0505730Z + DOCKER_SHELL_CMD= 2025-12-04T09:36:06.0506321Z + USED_IMAGE=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:36:06.0510661Z +++ nproc --ignore=2 2025-12-04T09:36:06.0676515Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e VLLM_TEST_HUGGING_FACE_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=122g --memory-swap=125g --env-file=/tmp/github_env_19923066595 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:36:19.3578562Z + container_name=b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:36:19.3579128Z + echo DOCKER_CONTAINER_ID=b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:36:19.3579532Z + [[ linux-jammy-py3.10-gcc11-build == *\s\3\9\0\x* ]] 2025-12-04T09:36:19.3584162Z ++ echo dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl 2025-12-04T09:36:19.3588079Z + docker exec -t b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 sh -c 'python3 -m pip install dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-12-04T09:36:19.8076831Z Processing ./dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl (from torch==2.10.0a0+gitffd9b0f) 2025-12-04T09:36:20.0397157Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.18.0) 2025-12-04T09:36:20.0401164Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (4.15.0) 2025-12-04T09:36:20.0402084Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.13.3) 2025-12-04T09:36:20.0408778Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (2.8.8) 2025-12-04T09:36:20.0414536Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.1.6) 2025-12-04T09:36:20.0415561Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (2025.10.0) 2025-12-04T09:36:20.0422859Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.3.0) 2025-12-04T09:36:20.0719809Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.22.4) 2025-12-04T09:36:20.0737234Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.3.0) 2025-12-04T09:36:20.0784587Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.0.3) 2025-12-04T09:36:20.8529822Z Installing collected packages: torch 2025-12-04T09:36:28.7313493Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-12-04T09:36:28.7314165Z dall-e 0.1 requires torchvision, which is not installed. 2025-12-04T09:36:28.7314483Z effdet 0.4.1 requires torchvision, which is not installed. 2025-12-04T09:36:28.7314828Z python-doctr 1.0.0 requires torchvision>=0.15.0, which is not installed. 2025-12-04T09:36:28.7315243Z pytorch-labs-segment-anything-fast 0.2 requires torchao, which is not installed. 2025-12-04T09:36:28.7315745Z pytorch-labs-segment-anything-fast 0.2 requires torchvision>=0.17.0.dev20231026, which is not installed. 2025-12-04T09:36:28.7316262Z timm 1.0.22 requires torchvision, which is not installed. 2025-12-04T09:36:28.7316635Z Successfully installed torch-2.10.0a0+gitffd9b0f 2025-12-04T09:36:28.8435909Z + export TERM=vt100 2025-12-04T09:36:28.8436163Z + TERM=vt100 2025-12-04T09:36:28.8437654Z ++ dirname .ci/pytorch/test.sh 2025-12-04T09:36:28.8452496Z + source .ci/pytorch/common.sh 2025-12-04T09:36:28.8452883Z +++ dirname .ci/pytorch/common.sh 2025-12-04T09:36:28.8461376Z ++ source .ci/pytorch/common_utils.sh 2025-12-04T09:36:28.8464892Z +++ declare -f -t trap_add 2025-12-04T09:36:28.8465274Z ++ set -ex -o pipefail 2025-12-04T09:36:28.8465555Z ++ [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-12-04T09:36:28.8465829Z ++ BUILD_TEST_LIBTORCH=0 2025-12-04T09:36:28.8470431Z ++ dirname .ci/pytorch/test.sh 2025-12-04T09:36:28.8476084Z + source .ci/pytorch/common-build.sh 2025-12-04T09:36:28.8486635Z ++ [[ linux-jammy-py3.10-gcc11-build != *win-* ]] 2025-12-04T09:36:28.8491999Z ++++ dirname .ci/pytorch/common-build.sh 2025-12-04T09:36:28.8507036Z +++ cd .ci/pytorch 2025-12-04T09:36:28.8507292Z +++ pwd -P 2025-12-04T09:36:28.8517185Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-12-04T09:36:28.8517567Z ++ [[ linux-jammy-py3.10-gcc11-build == *-pch* ]] 2025-12-04T09:36:28.8517815Z ++ which sccache 2025-12-04T09:36:28.8533698Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-12-04T09:36:28.8534164Z ++ sccache --stop-server 2025-12-04T09:36:28.8559044Z ++ true 2025-12-04T09:36:28.8559286Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-12-04T09:36:28.8573381Z ++ trap_add sccache_epilogue EXIT 2025-12-04T09:36:28.8573759Z ++ trap_add_cmd=sccache_epilogue 2025-12-04T09:36:28.8574141Z ++ shift 2025-12-04T09:36:28.8574377Z ++ for trap_add_name in "$@" 2025-12-04T09:36:28.8582627Z ++++ trap -p EXIT 2025-12-04T09:36:28.8582873Z +++ eval 'extract_trap_cmd ' 2025-12-04T09:36:28.8583092Z ++++ extract_trap_cmd 2025-12-04T09:36:28.8583278Z ++++ printf '%s\n' '' 2025-12-04T09:36:28.8583492Z +++ printf '%s\n' sccache_epilogue 2025-12-04T09:36:28.8586733Z ++ trap -- ' 2025-12-04T09:36:28.8587165Z sccache_epilogue' EXIT 2025-12-04T09:36:28.8587347Z ++ [[ -n 1 ]] 2025-12-04T09:36:28.8587629Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-12-04T09:36:28.8588045Z Skipping sccache server initialization, setting environment variables 2025-12-04T09:36:28.8588360Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:36:28.8588571Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:36:28.8588826Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:36:28.8589143Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:36:28.8596980Z ++ export RUST_LOG=sccache::server=error 2025-12-04T09:36:28.8597265Z ++ RUST_LOG=sccache::server=error 2025-12-04T09:36:28.8597498Z ++ sccache --zero-stats 2025-12-04T09:36:28.9999905Z Statistics zeroed. 2025-12-04T09:36:29.0009372Z ++ which ccache 2025-12-04T09:36:29.0036536Z + [[ linux-jammy-py3.10-gcc11-build != *rocm* ]] 2025-12-04T09:36:29.0037043Z + [[ linux-jammy-py3.10-gcc11-build != *s390x* ]] 2025-12-04T09:36:29.0037334Z + [[ -d /var/lib/jenkins/workspace ]] 2025-12-04T09:36:29.0038517Z ++ stat -c %u /var/lib/jenkins/workspace 2025-12-04T09:36:29.0061233Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-12-04T09:36:29.0061543Z + trap_add cleanup_workspace EXIT 2025-12-04T09:36:29.0061807Z + trap_add_cmd=cleanup_workspace 2025-12-04T09:36:29.0062013Z + shift 2025-12-04T09:36:29.0062204Z + for trap_add_name in "$@" 2025-12-04T09:36:29.0062418Z +++ trap -p EXIT 2025-12-04T09:36:29.0067389Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-12-04T09:36:29.0067700Z sccache_epilogue'\'' EXIT' 2025-12-04T09:36:29.0067927Z +++ extract_trap_cmd trap -- ' 2025-12-04T09:36:29.0068150Z sccache_epilogue' EXIT 2025-12-04T09:36:29.0068337Z +++ printf '%s\n' ' 2025-12-04T09:36:29.0068528Z sccache_epilogue' 2025-12-04T09:36:29.0068737Z ++ printf '%s\n' cleanup_workspace 2025-12-04T09:36:29.0069021Z + trap -- ' 2025-12-04T09:36:29.0069192Z sccache_epilogue 2025-12-04T09:36:29.0069375Z cleanup_workspace' EXIT 2025-12-04T09:36:29.0069608Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-12-04T09:36:29.4507752Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-12-04T09:36:29.4517274Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T09:36:29.4518076Z + echo 'Environment variables:' 2025-12-04T09:36:29.4518455Z Environment variables: 2025-12-04T09:36:29.4518666Z + env 2025-12-04T09:36:29.4528656Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:36:29.4529040Z CONTINUE_THROUGH_ERROR=True 2025-12-04T09:36:29.4529322Z BUILD_ENVIRONMENT=linux-jammy-py3.10-gcc11-build 2025-12-04T09:36:29.4529799Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-12-04T09:36:29.4530018Z HOSTNAME=b931eaab4b96 2025-12-04T09:36:29.4530431Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_cf740c30-289e-4d9c-945b-aa712ba26989 2025-12-04T09:36:29.4531068Z GITHUB_ACTION=__run_3 2025-12-04T09:36:29.4531274Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-12-04T09:36:29.4531506Z GITHUB_RUN_NUMBER=67027 2025-12-04T09:36:29.4531719Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-12-04T09:36:29.4531965Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-12-04T09:36:29.4532213Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-12-04T09:36:29.4532448Z SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:36:29.4532770Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-12-04T09:36:29.4533003Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-12-04T09:36:29.4533265Z GITHUB_REF_TYPE=branch 2025-12-04T09:36:29.4533493Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:36:29.4533732Z XLA_CUDA= 2025-12-04T09:36:29.4533900Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-12-04T09:36:29.4534197Z HUGGING_FACE_HUB_TOKEN=*** 2025-12-04T09:36:29.4534663Z *** 2025-12-04T09:36:29.4534837Z GITHUB_REPOSITORY_ID=65600975 2025-12-04T09:36:29.4535040Z GITHUB_ACTIONS=true 2025-12-04T09:36:29.4535265Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:36:29.4535553Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:36:29.4535825Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:36:29.4536217Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/heads/main 2025-12-04T09:36:29.4536622Z UCC_HOME=/usr 2025-12-04T09:36:29.4536803Z VERBOSE_TEST_LOGS=False 2025-12-04T09:36:29.4536994Z GITHUB_REF=refs/heads/main 2025-12-04T09:36:29.4537191Z SHARD_NUMBER=1 2025-12-04T09:36:29.4537373Z GITHUB_REF_PROTECTED=true 2025-12-04T09:36:29.4537564Z HOME=/var/lib/jenkins 2025-12-04T09:36:29.4537778Z GITHUB_API_URL=https://api.github.com 2025-12-04T09:36:29.4538023Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-12-04T09:36:29.4538231Z UCX_COMMIT= 2025-12-04T09:36:29.4538396Z USE_SYSTEM_NCCL=1 2025-12-04T09:36:29.4538573Z NUM_TEST_SHARDS=1 2025-12-04T09:36:29.4538736Z UCX_HOME=/usr 2025-12-04T09:36:29.4539121Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_cf740c30-289e-4d9c-945b-aa712ba26989 2025-12-04T09:36:29.4539739Z JOB_NAME=periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:36:29.4540342Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_cf740c30-289e-4d9c-945b-aa712ba26989 2025-12-04T09:36:29.4540857Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-12-04T09:36:29.4541193Z GITHUB_EVENT_NAME=schedule 2025-12-04T09:36:29.4541390Z DASHBOARD_TAG= 2025-12-04T09:36:29.4541558Z GITHUB_RUN_ID=19923066595 2025-12-04T09:36:29.4541752Z INSTALLED_OPENBLAS= 2025-12-04T09:36:29.4542162Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_cf740c30-289e-4d9c-945b-aa712ba26989 2025-12-04T09:36:29.4542603Z GITHUB_ACTOR=pytorchmergebot 2025-12-04T09:36:29.4542799Z PR_NUMBER= 2025-12-04T09:36:29.4542963Z DESIRED_CUDA= 2025-12-04T09:36:29.4543133Z GITHUB_RUN_ATTEMPT=1 2025-12-04T09:36:29.4543322Z ANACONDA_PYTHON_VERSION=3.10 2025-12-04T09:36:29.4543566Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-12-04T09:36:29.4543812Z TERM=vt100 2025-12-04T09:36:29.4543970Z INSTALLED_VISION=yes 2025-12-04T09:36:29.4544147Z BRANCH=main 2025-12-04T09:36:29.4544317Z SCCACHE_REGION=us-east-1 2025-12-04T09:36:29.4544517Z OPENSSL_ROOT_DIR=/opt/openssl 2025-12-04T09:36:29.4544802Z BUILD_AOT_INDUCTOR_TEST=yes 2025-12-04T09:36:29.4545014Z CUDA_PATH=/usr/local/cuda 2025-12-04T09:36:29.4545364Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-12-04T09:36:29.4545772Z GITHUB_SERVER_URL=https://github.com 2025-12-04T09:36:29.4545991Z UCC_COMMIT= 2025-12-04T09:36:29.4546157Z REENABLED_ISSUES= 2025-12-04T09:36:29.4546320Z DOCS=yes 2025-12-04T09:36:29.4546475Z SHLVL=1 2025-12-04T09:36:29.4546627Z MAX_JOBS=30 2025-12-04T09:36:29.4546783Z GITHUB_ACTOR_ID=97764156 2025-12-04T09:36:29.4547033Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:36:29.4547536Z GITHUB_REF_NAME=main 2025-12-04T09:36:29.4547812Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:36:29.4548119Z GITHUB_JOB=test 2025-12-04T09:36:29.4548292Z NO_TEST_TIMEOUT=False 2025-12-04T09:36:29.4548466Z TD_DISTRIBUTED=False 2025-12-04T09:36:29.4548663Z GITHUB_REPOSITORY=pytorch/pytorch 2025-12-04T09:36:29.4548885Z GITHUB_RETENTION_DAYS=90 2025-12-04T09:36:29.4549069Z OPENSSL_DIR=/opt/openssl 2025-12-04T09:36:29.4549263Z GITHUB_ACTION_REPOSITORY= 2025-12-04T09:36:29.4549878Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:36:29.4550377Z GITHUB_BASE_REF= 2025-12-04T09:36:29.4550543Z INSTALLED_ACL= 2025-12-04T09:36:29.4550865Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290 2025-12-04T09:36:29.4551214Z CI=true 2025-12-04T09:36:29.4551376Z GITHUB_REPOSITORY_OWNER=pytorch 2025-12-04T09:36:29.4551641Z RUST_LOG=sccache::server=error 2025-12-04T09:36:29.4551841Z JOB_ID=57118563290 2025-12-04T09:36:29.4552003Z GITHUB_HEAD_REF= 2025-12-04T09:36:29.4552174Z GITHUB_ACTION_REF= 2025-12-04T09:36:29.4552386Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-12-04T09:36:29.4552622Z TEST_SHOWLOCALS=False 2025-12-04T09:36:29.4552826Z GITHUB_WORKFLOW=inductor-periodic 2025-12-04T09:36:29.4553045Z DEBIAN_FRONTEND=noninteractive 2025-12-04T09:36:29.4553450Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_cf740c30-289e-4d9c-945b-aa712ba26989 2025-12-04T09:36:29.4553842Z NO_TD=False 2025-12-04T09:36:29.4554014Z SKIP_SCCACHE_INITIALIZATION=1 2025-12-04T09:36:29.4554231Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-12-04T09:36:29.4554438Z _=/usr/bin/env 2025-12-04T09:36:29.4554669Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-12-04T09:36:29.4805955Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2025-12-04T09:36:29.4811066Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-12-04T09:36:29.4815497Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2025-12-04T09:36:29.4818133Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2025-12-04T09:36:29.4818586Z + BUILD_DIR=build 2025-12-04T09:36:29.4818836Z + BUILD_RENAMED_DIR=build_renamed 2025-12-04T09:36:29.4819055Z + BUILD_BIN_DIR=build/bin 2025-12-04T09:36:29.4819256Z + SHARD_NUMBER=1 2025-12-04T09:36:29.4819438Z + NUM_TEST_SHARDS=1 2025-12-04T09:36:29.4819638Z + export TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:36:29.4819876Z + TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:36:29.4820084Z + export VALGRIND=ON 2025-12-04T09:36:29.4820263Z + VALGRIND=ON 2025-12-04T09:36:29.4820477Z + [[ linux-jammy-py3.10-gcc11-build == *clang9* ]] 2025-12-04T09:36:29.4820762Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-12-04T09:36:29.4821006Z + detect_cuda_arch 2025-12-04T09:36:29.4821222Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T09:36:29.4821534Z + [[ linux-jammy-py3.10-gcc11-build == *s390x* ]] 2025-12-04T09:36:29.4821759Z + [[ 0 == \1 ]] 2025-12-04T09:36:29.4821930Z + [[ True == \1 ]] 2025-12-04T09:36:29.4822139Z + [[ linux-jammy-py3.10-gcc11-build != *bazel* ]] 2025-12-04T09:36:29.4822407Z ++ realpath build/custom_test_artifacts 2025-12-04T09:36:29.4823037Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-12-04T09:36:29.4823380Z + [[ -n '' ]] 2025-12-04T09:36:29.4823568Z + echo 'Environment variables' 2025-12-04T09:36:29.4823777Z Environment variables 2025-12-04T09:36:29.4823965Z + env 2025-12-04T09:36:29.4841781Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:36:29.4842093Z CONTINUE_THROUGH_ERROR=True 2025-12-04T09:36:29.4842338Z BUILD_ENVIRONMENT=linux-jammy-py3.10-gcc11-build 2025-12-04T09:36:29.4842812Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-12-04T09:36:29.4843153Z HOSTNAME=b931eaab4b96 2025-12-04T09:36:29.4843549Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_cf740c30-289e-4d9c-945b-aa712ba26989 2025-12-04T09:36:29.4843970Z GITHUB_ACTION=__run_3 2025-12-04T09:36:29.4844177Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-12-04T09:36:29.4844406Z GITHUB_RUN_NUMBER=67027 2025-12-04T09:36:29.4844631Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-12-04T09:36:29.4844880Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-12-04T09:36:29.4845122Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-12-04T09:36:29.4845345Z SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:36:29.4845650Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-12-04T09:36:29.4845876Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-12-04T09:36:29.4846091Z GITHUB_REF_TYPE=branch 2025-12-04T09:36:29.4846320Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:36:29.4846567Z XLA_CUDA= 2025-12-04T09:36:29.4846741Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-12-04T09:36:29.4847279Z HUGGING_FACE_HUB_TOKEN=*** 2025-12-04T09:36:29.4847548Z *** 2025-12-04T09:36:29.4847717Z GITHUB_REPOSITORY_ID=65600975 2025-12-04T09:36:29.4847920Z GITHUB_ACTIONS=true 2025-12-04T09:36:29.4848189Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:36:29.4848468Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:36:29.4848739Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:36:29.4849127Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/heads/main 2025-12-04T09:36:29.4849588Z UCC_HOME=/usr 2025-12-04T09:36:29.4849771Z TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:36:29.4849976Z VERBOSE_TEST_LOGS=False 2025-12-04T09:36:29.4850163Z GITHUB_REF=refs/heads/main 2025-12-04T09:36:29.4850352Z SHARD_NUMBER=1 2025-12-04T09:36:29.4850530Z GITHUB_REF_PROTECTED=true 2025-12-04T09:36:29.4850718Z HOME=/var/lib/jenkins 2025-12-04T09:36:29.4850933Z GITHUB_API_URL=https://api.github.com 2025-12-04T09:36:29.4851176Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-12-04T09:36:29.4851382Z UCX_COMMIT= 2025-12-04T09:36:29.4851551Z USE_SYSTEM_NCCL=1 2025-12-04T09:36:29.4851725Z NUM_TEST_SHARDS=1 2025-12-04T09:36:29.4851886Z UCX_HOME=/usr 2025-12-04T09:36:29.4852270Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_cf740c30-289e-4d9c-945b-aa712ba26989 2025-12-04T09:36:29.4852889Z JOB_NAME=periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:36:29.4853480Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_cf740c30-289e-4d9c-945b-aa712ba26989 2025-12-04T09:36:29.4853994Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-12-04T09:36:29.4854325Z GITHUB_EVENT_NAME=schedule 2025-12-04T09:36:29.4854524Z DASHBOARD_TAG= 2025-12-04T09:36:29.4854693Z GITHUB_RUN_ID=19923066595 2025-12-04T09:36:29.4854892Z INSTALLED_OPENBLAS= 2025-12-04T09:36:29.4855308Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_cf740c30-289e-4d9c-945b-aa712ba26989 2025-12-04T09:36:29.4855750Z GITHUB_ACTOR=pytorchmergebot 2025-12-04T09:36:29.4855959Z PR_NUMBER= 2025-12-04T09:36:29.4856124Z DESIRED_CUDA= 2025-12-04T09:36:29.4856287Z GITHUB_RUN_ATTEMPT=1 2025-12-04T09:36:29.4856468Z VALGRIND=ON 2025-12-04T09:36:29.4856643Z ANACONDA_PYTHON_VERSION=3.10 2025-12-04T09:36:29.4857020Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-12-04T09:36:29.4857263Z TERM=vt100 2025-12-04T09:36:29.4857460Z INSTALLED_VISION=yes 2025-12-04T09:36:29.4857643Z BRANCH=main 2025-12-04T09:36:29.4857808Z SCCACHE_REGION=us-east-1 2025-12-04T09:36:29.4858016Z OPENSSL_ROOT_DIR=/opt/openssl 2025-12-04T09:36:29.4858231Z BUILD_AOT_INDUCTOR_TEST=yes 2025-12-04T09:36:29.4858427Z CUDA_PATH=/usr/local/cuda 2025-12-04T09:36:29.4858792Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-12-04T09:36:29.4859168Z GITHUB_SERVER_URL=https://github.com 2025-12-04T09:36:29.4859376Z UCC_COMMIT= 2025-12-04T09:36:29.4859652Z REENABLED_ISSUES= 2025-12-04T09:36:29.4859824Z DOCS=yes 2025-12-04T09:36:29.4859976Z SHLVL=1 2025-12-04T09:36:29.4860136Z MAX_JOBS=30 2025-12-04T09:36:29.4860306Z GITHUB_ACTOR_ID=97764156 2025-12-04T09:36:29.4860553Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:36:29.4860828Z GITHUB_REF_NAME=main 2025-12-04T09:36:29.4861109Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:36:29.4861404Z GITHUB_JOB=test 2025-12-04T09:36:29.4861576Z NO_TEST_TIMEOUT=False 2025-12-04T09:36:29.4861763Z TD_DISTRIBUTED=False 2025-12-04T09:36:29.4861962Z GITHUB_REPOSITORY=pytorch/pytorch 2025-12-04T09:36:29.4862176Z GITHUB_RETENTION_DAYS=90 2025-12-04T09:36:29.4862371Z OPENSSL_DIR=/opt/openssl 2025-12-04T09:36:29.4862569Z GITHUB_ACTION_REPOSITORY= 2025-12-04T09:36:29.4863067Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:36:29.4863570Z GITHUB_BASE_REF= 2025-12-04T09:36:29.4863746Z INSTALLED_ACL= 2025-12-04T09:36:29.4864061Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290 2025-12-04T09:36:29.4864406Z CI=true 2025-12-04T09:36:29.4864579Z GITHUB_REPOSITORY_OWNER=pytorch 2025-12-04T09:36:29.4864846Z RUST_LOG=sccache::server=error 2025-12-04T09:36:29.4865047Z JOB_ID=57118563290 2025-12-04T09:36:29.4865219Z GITHUB_HEAD_REF= 2025-12-04T09:36:29.4865393Z GITHUB_ACTION_REF= 2025-12-04T09:36:29.4865604Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-12-04T09:36:29.4865855Z TEST_SHOWLOCALS=False 2025-12-04T09:36:29.4866057Z GITHUB_WORKFLOW=inductor-periodic 2025-12-04T09:36:29.4866278Z DEBIAN_FRONTEND=noninteractive 2025-12-04T09:36:29.4866690Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_cf740c30-289e-4d9c-945b-aa712ba26989 2025-12-04T09:36:29.4867102Z NO_TD=False 2025-12-04T09:36:29.4867275Z SKIP_SCCACHE_INITIALIZATION=1 2025-12-04T09:36:29.4867507Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-12-04T09:36:29.4867730Z _=/usr/bin/env 2025-12-04T09:36:29.4867897Z + echo 'Testing pytorch' 2025-12-04T09:36:29.4868089Z Testing pytorch 2025-12-04T09:36:29.4868278Z + export LANG=C.UTF-8 2025-12-04T09:36:29.4868456Z + LANG=C.UTF-8 2025-12-04T09:36:29.4868625Z + PR_NUMBER= 2025-12-04T09:36:29.4868844Z + [[ dynamic_cpu_inductor_huggingface == \d\e\f\a\u\l\t ]] 2025-12-04T09:36:29.4869160Z + [[ dynamic_cpu_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-12-04T09:36:29.4869455Z + [[ dynamic_cpu_inductor_huggingface == \s\l\o\w ]] 2025-12-04T09:36:29.4869751Z + [[ linux-jammy-py3.10-gcc11-build == *slow-gradcheck* ]] 2025-12-04T09:36:29.4870042Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T09:36:29.4870300Z + [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-12-04T09:36:29.4870563Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-12-04T09:36:29.4870837Z + [[ dynamic_cpu_inductor_huggingface == *crossref* ]] 2025-12-04T09:36:29.4871104Z + [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-12-04T09:36:29.4871360Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-12-04T09:36:29.4871627Z + [[ linux-jammy-py3.10-gcc11-build != *-bazel-* ]] 2025-12-04T09:36:29.4871878Z + pip_install ninja==1.10.2 2025-12-04T09:36:29.4872139Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-12-04T09:36:29.4872506Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-12-04T09:36:29.8989323Z Collecting ninja==1.10.2 2025-12-04T09:36:29.9212312Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-12-04T09:36:29.9337952Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-12-04T09:36:30.7855966Z Installing collected packages: ninja 2025-12-04T09:36:30.7856413Z Attempting uninstall: ninja 2025-12-04T09:36:30.7863751Z Found existing installation: ninja 1.11.1.4 2025-12-04T09:36:30.7879496Z Uninstalling ninja-1.11.1.4: 2025-12-04T09:36:30.8121077Z Successfully uninstalled ninja-1.11.1.4 2025-12-04T09:36:30.8757901Z Successfully installed ninja-1.10.2 2025-12-04T09:36:30.9894820Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:36:30.9895911Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:36:30.9896537Z + [[ linux-jammy-py3.10-gcc11-build == *aarch64* ]] 2025-12-04T09:36:30.9896821Z + [[ linux-jammy-py3.10-gcc11-build == *asan* ]] 2025-12-04T09:36:30.9897103Z + [[ linux-jammy-py3.10-gcc11-build == *-debug* ]] 2025-12-04T09:36:30.9897391Z + [[ linux-jammy-py3.10-gcc11-build != *-bazel-* ]] 2025-12-04T09:36:30.9897799Z + echo 'We are not in debug mode: linux-jammy-py3.10-gcc11-build. Expect the assertion to pass' 2025-12-04T09:36:30.9898274Z We are not in debug mode: linux-jammy-py3.10-gcc11-build. Expect the assertion to pass 2025-12-04T09:36:30.9898592Z + cd test 2025-12-04T09:36:30.9898853Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-12-04T09:36:31.3059447Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:36:31.3060328Z import pynvml # type: ignore[import] 2025-12-04T09:36:32.2526944Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-12-04T09:36:32.2527387Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-12-04T09:36:32.2527774Z + [[ dynamic_cpu_inductor_huggingface == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-12-04T09:36:32.2528992Z + DYNAMO_BENCHMARK_FLAGS=() 2025-12-04T09:36:32.2529313Z + [[ dynamic_cpu_inductor_huggingface == *pr_time_benchmarks* ]] 2025-12-04T09:36:32.2529637Z + [[ dynamic_cpu_inductor_huggingface == *dynamo_eager* ]] 2025-12-04T09:36:32.2529930Z + [[ dynamic_cpu_inductor_huggingface == *aot_eager* ]] 2025-12-04T09:36:32.2530215Z + [[ dynamic_cpu_inductor_huggingface == *aot_inductor* ]] 2025-12-04T09:36:32.2530569Z + [[ dynamic_cpu_inductor_huggingface == *max_autotune_inductor* ]] 2025-12-04T09:36:32.2530881Z + [[ dynamic_cpu_inductor_huggingface == *inductor* ]] 2025-12-04T09:36:32.2531147Z + [[ dynamic_cpu_inductor_huggingface != *perf* ]] 2025-12-04T09:36:32.2531433Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2025-12-04T09:36:32.2531698Z + [[ dynamic_cpu_inductor_huggingface == *dynamic* ]] 2025-12-04T09:36:32.2532020Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2025-12-04T09:36:32.2532337Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-12-04T09:36:32.2532590Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu) 2025-12-04T09:36:32.2780751Z + [[ linux-jammy-py3.10-gcc11-build == *libtorch* ]] 2025-12-04T09:36:32.2783936Z + [[ linux-jammy-py3.10-gcc11-build == *-bazel-* ]] 2025-12-04T09:36:32.2788418Z + cd test 2025-12-04T09:36:32.2788758Z + python -c 'import torch; print(torch.__config__.show())' 2025-12-04T09:36:32.5848716Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:36:32.5850169Z import pynvml # type: ignore[import] 2025-12-04T09:36:33.2808133Z PyTorch built with: 2025-12-04T09:36:33.2808407Z - GCC 11.4 2025-12-04T09:36:33.2808577Z - C++ Version: 201703 2025-12-04T09:36:33.2808972Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-12-04T09:36:33.2809471Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-12-04T09:36:33.2810172Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-12-04T09:36:33.2810426Z - LAPACK is enabled (usually provided by MKL) 2025-12-04T09:36:33.2810662Z - NNPACK is enabled 2025-12-04T09:36:33.2810857Z - CPU capability usage: AVX512 2025-12-04T09:36:33.2813923Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.10.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-12-04T09:36:33.2816873Z 2025-12-04T09:36:33.5196700Z + cd test 2025-12-04T09:36:33.5197059Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-12-04T09:36:33.8116505Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:36:33.8117923Z import pynvml # type: ignore[import] 2025-12-04T09:36:34.5136067Z ATen/Parallel: 2025-12-04T09:36:34.5136385Z at::get_num_threads() : 16 2025-12-04T09:36:34.5136626Z at::get_num_interop_threads() : 16 2025-12-04T09:36:34.5136864Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-12-04T09:36:34.5137109Z omp_get_max_threads() : 16 2025-12-04T09:36:34.5137547Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-12-04T09:36:34.5137947Z mkl_get_max_threads() : 16 2025-12-04T09:36:34.5138236Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-12-04T09:36:34.5138558Z std::thread::hardware_concurrency() : 32 2025-12-04T09:36:34.5138801Z Environment variables: 2025-12-04T09:36:34.5139000Z OMP_NUM_THREADS : [not set] 2025-12-04T09:36:34.5139196Z MKL_NUM_THREADS : [not set] 2025-12-04T09:36:34.5139403Z ATen parallel backend: OpenMP 2025-12-04T09:36:34.5139535Z 2025-12-04T09:36:34.7530954Z + [[ dynamic_cpu_inductor_huggingface == *numpy_2* ]] 2025-12-04T09:36:34.7531418Z + [[ linux-jammy-py3.10-gcc11-build == *aarch64* ]] 2025-12-04T09:36:34.7531714Z + [[ dynamic_cpu_inductor_huggingface == *backward* ]] 2025-12-04T09:36:34.7532058Z + [[ dynamic_cpu_inductor_huggingface == *libtorch_agnostic_targetting* ]] 2025-12-04T09:36:34.7532397Z + [[ dynamic_cpu_inductor_huggingface == *xla* ]] 2025-12-04T09:36:34.7533013Z + [[ dynamic_cpu_inductor_huggingface == *vllm* ]] 2025-12-04T09:36:34.7533303Z + [[ dynamic_cpu_inductor_huggingface == *executorch* ]] 2025-12-04T09:36:34.7533611Z + [[ dynamic_cpu_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2025-12-04T09:36:34.7533949Z + [[ dynamic_cpu_inductor_huggingface == \q\u\a\n\t\i\z\a\t\i\o\n ]] 2025-12-04T09:36:34.7534268Z + [[ linux-jammy-py3.10-gcc11-build == *libtorch* ]] 2025-12-04T09:36:34.7534556Z + [[ dynamic_cpu_inductor_huggingface == distributed ]] 2025-12-04T09:36:34.7534864Z + [[ dynamic_cpu_inductor_huggingface == *operator_benchmark* ]] 2025-12-04T09:36:34.7535202Z + [[ dynamic_cpu_inductor_huggingface == *operator_microbenchmark* ]] 2025-12-04T09:36:34.7535681Z + [[ dynamic_cpu_inductor_huggingface == *attention_microbenchmark* ]] 2025-12-04T09:36:34.7536034Z + [[ dynamic_cpu_inductor_huggingface == *inductor_distributed* ]] 2025-12-04T09:36:34.7536360Z + [[ dynamic_cpu_inductor_huggingface == *inductor-halide* ]] 2025-12-04T09:36:34.7536658Z + [[ dynamic_cpu_inductor_huggingface == *inductor-pallas* ]] 2025-12-04T09:36:34.7536979Z + [[ dynamic_cpu_inductor_huggingface == *inductor-triton-cpu* ]] 2025-12-04T09:36:34.7537335Z + [[ dynamic_cpu_inductor_huggingface == *inductor-micro-benchmark* ]] 2025-12-04T09:36:34.7537697Z + [[ dynamic_cpu_inductor_huggingface == *aoti_cross_compile_for_windows* ]] 2025-12-04T09:36:34.7538090Z + [[ dynamic_cpu_inductor_huggingface == *huggingface* ]] 2025-12-04T09:36:34.7538342Z + install_torchvision 2025-12-04T09:36:34.7538524Z + local orig_preload 2025-12-04T09:36:34.7538709Z + local commit 2025-12-04T09:36:34.7538888Z ++ get_pinned_commit vision 2025-12-04T09:36:34.7539103Z ++ cat .github/ci_commit_pins/vision.txt 2025-12-04T09:36:34.7544839Z + commit=617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T09:36:34.7545096Z + orig_preload= 2025-12-04T09:36:34.7545276Z + '[' -n '' ']' 2025-12-04T09:36:34.7545471Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T09:36:34.7545942Z + pip_build_and_install git+https://github.com/pytorch/vision.git@617079d944b0e72632311c30ae2bbdf1168b901e dist/vision 2025-12-04T09:36:34.7546508Z + local build_target=git+https://github.com/pytorch/vision.git@617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T09:36:34.7546885Z + local wheel_dir=dist/vision 2025-12-04T09:36:34.7547208Z + local found_whl=0 2025-12-04T09:36:34.7547408Z + for file in "${wheel_dir}"/*.whl 2025-12-04T09:36:34.7547733Z + [[ -f dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl ]] 2025-12-04T09:36:34.7548050Z + found_whl=1 2025-12-04T09:36:34.7548218Z + break 2025-12-04T09:36:34.7548378Z + '[' 1 == 0 ']' 2025-12-04T09:36:34.7548555Z + for file in "${wheel_dir}"/*.whl 2025-12-04T09:36:34.7548945Z + pip_install_whl dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:36:34.7549447Z + args=('dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl') 2025-12-04T09:36:34.7549753Z + local args 2025-12-04T09:36:34.7550032Z + [[ dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl == *\ * ]] 2025-12-04T09:36:34.7550361Z + for path in "${args[@]}" 2025-12-04T09:36:34.7550686Z + echo 'Installing dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl' 2025-12-04T09:36:34.7551138Z Installing dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:36:34.7551652Z + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:36:35.0566014Z Processing ./dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:36:35.0638695Z Installing collected packages: torchvision 2025-12-04T09:36:35.4614906Z Successfully installed torchvision-0.25.0a0+617079d 2025-12-04T09:36:35.5046422Z + '[' -n '' ']' 2025-12-04T09:36:35.5051470Z + id=0 2025-12-04T09:36:35.5055358Z + test_dynamo_benchmark huggingface 0 2025-12-04T09:36:35.5057609Z ++ pwd 2025-12-04T09:36:35.5058045Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-12-04T09:36:35.5058522Z + local suite=huggingface 2025-12-04T09:36:35.5059130Z + shift 2025-12-04T09:36:35.5059296Z + local shard_id=0 2025-12-04T09:36:35.5059465Z + shift 2025-12-04T09:36:35.5059623Z + extra_args=() 2025-12-04T09:36:35.5059796Z + local extra_args 2025-12-04T09:36:35.5060000Z + [[ linux-jammy-py3.10-gcc11-build == *cuda13* ]] 2025-12-04T09:36:35.5060301Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-12-04T09:36:35.5060585Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-12-04T09:36:35.5060843Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-12-04T09:36:35.5061071Z + local dt=float32 2025-12-04T09:36:35.5061271Z + [[ dynamic_cpu_inductor_huggingface == *amp* ]] 2025-12-04T09:36:35.5061664Z + [[ dynamic_cpu_inductor_huggingface == *freezing* ]] 2025-12-04T09:36:35.5061997Z + test_single_dynamo_benchmark inference huggingface 0 --inference --float32 2025-12-04T09:36:35.5062293Z ++ pwd 2025-12-04T09:36:35.5062522Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-12-04T09:36:35.5062847Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-12-04T09:36:35.5076551Z + local name=inference 2025-12-04T09:36:35.5076985Z + shift 2025-12-04T09:36:35.5077201Z + local suite=huggingface 2025-12-04T09:36:35.5077398Z + shift 2025-12-04T09:36:35.5077558Z + local shard_id=0 2025-12-04T09:36:35.5077732Z + shift 2025-12-04T09:36:35.5078030Z + partition_flags=() 2025-12-04T09:36:35.5078252Z + local partition_flags 2025-12-04T09:36:35.5078459Z + [[ -n 1 ]] 2025-12-04T09:36:35.5078630Z + [[ -n 0 ]] 2025-12-04T09:36:35.5078933Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-12-04T09:36:35.5079364Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-12-04T09:36:35.5079655Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-12-04T09:36:35.5079924Z + [[ dynamic_cpu_inductor_huggingface == *_avx2* ]] 2025-12-04T09:36:35.5080204Z + [[ dynamic_cpu_inductor_huggingface == *_avx512* ]] 2025-12-04T09:36:35.5081461Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --print-compilation-time --inductor --dynamic-shapes --dynamic-batch-only --device cpu --inference --float32 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2025-12-04T09:36:36.2756003Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:36:36.2756938Z import pynvml # type: ignore[import] 2025-12-04T09:36:39.7708836Z 2025-12-04T09:36:39.7713368Z config.json: 0% 0.00/694 [00:00bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1696322Z 2025-12-04T09:38:11.1696442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1697014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1697553Z layer_outputs = layer_module( 2025-12-04T09:38:11.1697936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1698327Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1698775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1699226Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1699677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1700132Z self_outputs = self.self( 2025-12-04T09:38:11.1700571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.1701079Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1701633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1702278Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1702541Z 2025-12-04T09:38:11.1702653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1703212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1703734Z layer_outputs = layer_module( 2025-12-04T09:38:11.1704166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1704553Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1705003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1705466Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1705920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1706416Z self_outputs = self.self( 2025-12-04T09:38:11.1706845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.1707324Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1707856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1708501Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1708767Z 2025-12-04T09:38:11.1708881Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1709432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1709941Z layer_outputs = layer_module( 2025-12-04T09:38:11.1710314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1710706Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1711156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1711602Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1712053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1712494Z self_outputs = self.self( 2025-12-04T09:38:11.1712967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.1713460Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1714007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1714663Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1714921Z 2025-12-04T09:38:11.1715010Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1715245Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1715478Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1715705Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1715957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1716530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1717073Z layer_outputs = layer_module( 2025-12-04T09:38:11.1717452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1717858Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1718323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1718786Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1719292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1719761Z self_outputs = self.self( 2025-12-04T09:38:11.1720210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.1720788Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1721354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1721961Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.1722601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.1723140Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.1723336Z 2025-12-04T09:38:11.1723424Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1723696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1724272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1724814Z layer_outputs = layer_module( 2025-12-04T09:38:11.1725205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1725610Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1726076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1726535Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1727007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1727465Z self_outputs = self.self( 2025-12-04T09:38:11.1727901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.1728362Z attn_scores += diagonal_mask 2025-12-04T09:38:11.1728507Z 2025-12-04T09:38:11.1728622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1729198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1729733Z layer_outputs = layer_module( 2025-12-04T09:38:11.1730119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1730528Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1730991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1731454Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1731914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1732372Z self_outputs = self.self( 2025-12-04T09:38:11.1732812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.1733269Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.1733423Z 2025-12-04T09:38:11.1733539Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1734113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1734643Z layer_outputs = layer_module( 2025-12-04T09:38:11.1735023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1735475Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1735944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1736397Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1736859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1737314Z self_outputs = self.self( 2025-12-04T09:38:11.1737754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1738299Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1738901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1739575Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.1740055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.1740443Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.1740618Z 2025-12-04T09:38:11.1740735Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1741309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1741843Z layer_outputs = layer_module( 2025-12-04T09:38:11.1742236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1742657Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1743123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1743586Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1744049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1744507Z self_outputs = self.self( 2025-12-04T09:38:11.1744945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1745444Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1746026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1746629Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.1747403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.1747927Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.1748304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.1748689Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.1748855Z 2025-12-04T09:38:11.1748968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1749528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1750057Z layer_outputs = layer_module( 2025-12-04T09:38:11.1750439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1750828Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1751364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1751817Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1752263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1752705Z self_outputs = self.self( 2025-12-04T09:38:11.1753133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1753624Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1754241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1754856Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.1755084Z 2025-12-04T09:38:11.1755203Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1755769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1756300Z layer_outputs = layer_module( 2025-12-04T09:38:11.1756676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1757068Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1757523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1757976Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1758427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1758874Z self_outputs = self.self( 2025-12-04T09:38:11.1759307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1759794Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1760363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1761063Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.1761294Z 2025-12-04T09:38:11.1761420Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1761998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1762526Z layer_outputs = layer_module( 2025-12-04T09:38:11.1762906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1763292Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1763747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1764212Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1764675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1765129Z self_outputs = self.self( 2025-12-04T09:38:11.1765571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.1766161Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.1766427Z 2025-12-04T09:38:11.1766550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1767173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1767713Z layer_outputs = layer_module( 2025-12-04T09:38:11.1768102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1768506Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1768962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1769481Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1769949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.1770443Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.1770946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.1771421Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.1771575Z 2025-12-04T09:38:11.1771694Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1772208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1772699Z layer_outputs = layer_module( 2025-12-04T09:38:11.1773069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1773451Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1773868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.1774299Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.1774716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.1775122Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.1775538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.1776001Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.1776453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.1776891Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.1777037Z 2025-12-04T09:38:11.1777143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1777666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1778183Z layer_outputs = layer_module( 2025-12-04T09:38:11.1778544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1778914Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1779367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.1779822Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.1780251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.1780679Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.1781121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.1781579Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.1782084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.1782574Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.1782988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.1783352Z return self.act(input) 2025-12-04T09:38:11.1783483Z 2025-12-04T09:38:11.1783595Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1784143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1784671Z layer_outputs = layer_module( 2025-12-04T09:38:11.1785016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1785386Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1785811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.1786242Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.1786654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.1787082Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.1787524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.1787998Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.1788476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.1788929Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.1789077Z 2025-12-04T09:38:11.1789199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1789746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1790270Z layer_outputs = layer_module( 2025-12-04T09:38:11.1790649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1791036Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1791473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1791929Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1792376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1792815Z self_outputs = self.self( 2025-12-04T09:38:11.1793251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:38:11.1793704Z query_vectors = self.query(hidden_states) 2025-12-04T09:38:11.1793850Z 2025-12-04T09:38:11.1793968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1794510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1795028Z layer_outputs = layer_module( 2025-12-04T09:38:11.1795403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1795794Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1796233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1796714Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1797167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1797602Z self_outputs = self.self( 2025-12-04T09:38:11.1798031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.1798511Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1799054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1799712Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1799977Z 2025-12-04T09:38:11.1800089Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1800744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1801302Z layer_outputs = layer_module( 2025-12-04T09:38:11.1801690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1802087Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1802539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1802990Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1803436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1803880Z self_outputs = self.self( 2025-12-04T09:38:11.1804312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:38:11.1804763Z key_vectors = self.key(hidden_states) 2025-12-04T09:38:11.1804917Z 2025-12-04T09:38:11.1805029Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1805580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1806100Z layer_outputs = layer_module( 2025-12-04T09:38:11.1806470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1806845Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1807274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1807701Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1808121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1808554Z self_outputs = self.self( 2025-12-04T09:38:11.1808957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.1809420Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1809948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1810571Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1810829Z 2025-12-04T09:38:11.1810949Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1811498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1812051Z layer_outputs = layer_module( 2025-12-04T09:38:11.1812410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1812780Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1813213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1813662Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1814109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1814606Z self_outputs = self.self( 2025-12-04T09:38:11.1815024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.1815512Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1816019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1816607Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1816861Z 2025-12-04T09:38:11.1816975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1817521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1818035Z layer_outputs = layer_module( 2025-12-04T09:38:11.1818412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1818800Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1819246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1819697Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1820133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1820575Z self_outputs = self.self( 2025-12-04T09:38:11.1820999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.1821469Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1822003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1822631Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1822891Z 2025-12-04T09:38:11.1822980Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1823216Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1823436Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1823659Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1823915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1824492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1825013Z layer_outputs = layer_module( 2025-12-04T09:38:11.1825386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1825781Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1826224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1826679Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1827164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1827614Z self_outputs = self.self( 2025-12-04T09:38:11.1828037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.1828517Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1829061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1829640Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.1830243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.1830762Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.1830924Z 2025-12-04T09:38:11.1831019Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1831270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1831824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1832342Z layer_outputs = layer_module( 2025-12-04T09:38:11.1832716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1833101Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1833558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1834006Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1834444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1834887Z self_outputs = self.self( 2025-12-04T09:38:11.1835311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.1835759Z attn_scores += diagonal_mask 2025-12-04T09:38:11.1835890Z 2025-12-04T09:38:11.1836001Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1836551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1837071Z layer_outputs = layer_module( 2025-12-04T09:38:11.1837451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1837835Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1838285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1838734Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1839173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1839610Z self_outputs = self.self( 2025-12-04T09:38:11.1840030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.1840551Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.1840708Z 2025-12-04T09:38:11.1840825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1841414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1841964Z layer_outputs = layer_module( 2025-12-04T09:38:11.1842411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1842798Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1843246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1843684Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1844104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1844511Z self_outputs = self.self( 2025-12-04T09:38:11.1844916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:38:11.1845387Z value_vectors = self.value(hidden_states) 2025-12-04T09:38:11.1845528Z 2025-12-04T09:38:11.1845636Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1846166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1846660Z layer_outputs = layer_module( 2025-12-04T09:38:11.1847012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1847554Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1847981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1848406Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1848833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1849244Z self_outputs = self.self( 2025-12-04T09:38:11.1849650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1850123Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1850653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1851256Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.1851692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.1852055Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.1852212Z 2025-12-04T09:38:11.1852317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1852844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1853336Z layer_outputs = layer_module( 2025-12-04T09:38:11.1853694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1854058Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1854480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1854907Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1855332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1855815Z self_outputs = self.self( 2025-12-04T09:38:11.1856218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1856678Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1857278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1857834Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.1858355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.1858817Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.1859151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.1859501Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.1859705Z 2025-12-04T09:38:11.1859811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1860318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1860790Z layer_outputs = layer_module( 2025-12-04T09:38:11.1861140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1861504Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1861919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1862339Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1862765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1863176Z self_outputs = self.self( 2025-12-04T09:38:11.1863564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1864017Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1864537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1865098Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.1865299Z 2025-12-04T09:38:11.1865406Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1865927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1866420Z layer_outputs = layer_module( 2025-12-04T09:38:11.1866771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1867138Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1867565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1867990Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1868403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1868799Z self_outputs = self.self( 2025-12-04T09:38:11.1869196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1869649Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1870187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1870791Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.1871016Z 2025-12-04T09:38:11.1871129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1871735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1872217Z layer_outputs = layer_module( 2025-12-04T09:38:11.1872569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1872944Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1873368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1873802Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1874308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1874752Z self_outputs = self.self( 2025-12-04T09:38:11.1875177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.1875710Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.1875963Z 2025-12-04T09:38:11.1876077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1876635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1877158Z layer_outputs = layer_module( 2025-12-04T09:38:11.1877530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1877927Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1878381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1878826Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1879282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.1879766Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.1880253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.1880798Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.1880967Z 2025-12-04T09:38:11.1881083Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1881667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1882190Z layer_outputs = layer_module( 2025-12-04T09:38:11.1882561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1882958Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1883410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.1883875Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.1884314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.1884743Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.1885197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.1885688Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.1886180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.1886641Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.1886835Z 2025-12-04T09:38:11.1886958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1887503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1888005Z layer_outputs = layer_module( 2025-12-04T09:38:11.1888363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1888738Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1889162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.1889629Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.1890037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.1890435Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.1890875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.1891340Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.1891792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.1892245Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.1892635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.1892990Z return self.act(input) 2025-12-04T09:38:11.1893105Z 2025-12-04T09:38:11.1893219Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1893732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1894225Z layer_outputs = layer_module( 2025-12-04T09:38:11.1894574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1894933Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1895356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.1895782Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.1896192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.1896588Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.1897035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.1897535Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.1898034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.1898483Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.1898636Z 2025-12-04T09:38:11.1898748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1899306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1899825Z layer_outputs = layer_module( 2025-12-04T09:38:11.1900175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1900541Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1900964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1901439Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1901882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1902334Z self_outputs = self.self( 2025-12-04T09:38:11.1902742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:38:11.1903168Z query_vectors = self.query(hidden_states) 2025-12-04T09:38:11.1903314Z 2025-12-04T09:38:11.1903420Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1904010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1904503Z layer_outputs = layer_module( 2025-12-04T09:38:11.1904856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1905228Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1905654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1906075Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1906538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1906966Z self_outputs = self.self( 2025-12-04T09:38:11.1907378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.1907836Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1908351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1908952Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1909190Z 2025-12-04T09:38:11.1909302Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1909807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1910286Z layer_outputs = layer_module( 2025-12-04T09:38:11.1910629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1910990Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1911396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1911810Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1912230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1912704Z self_outputs = self.self( 2025-12-04T09:38:11.1913131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:38:11.1913582Z key_vectors = self.key(hidden_states) 2025-12-04T09:38:11.1913725Z 2025-12-04T09:38:11.1913844Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1914387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1914906Z layer_outputs = layer_module( 2025-12-04T09:38:11.1915284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1915673Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1916152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1916601Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1917044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1917483Z self_outputs = self.self( 2025-12-04T09:38:11.1917902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.1918420Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1918973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1919653Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1919947Z 2025-12-04T09:38:11.1920064Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1920735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1921281Z layer_outputs = layer_module( 2025-12-04T09:38:11.1921660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1922072Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1922542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1922998Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1923443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1923894Z self_outputs = self.self( 2025-12-04T09:38:11.1924331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.1924810Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1925343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1925972Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1926228Z 2025-12-04T09:38:11.1926351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1926901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1927413Z layer_outputs = layer_module( 2025-12-04T09:38:11.1927792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1928187Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1928629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1929080Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1929529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1929972Z self_outputs = self.self( 2025-12-04T09:38:11.1930395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.1930871Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1931464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1932094Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.1932352Z 2025-12-04T09:38:11.1932443Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1932680Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1932909Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1933125Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1933378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1933935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1934495Z layer_outputs = layer_module( 2025-12-04T09:38:11.1934866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1935264Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1935714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1936157Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1936602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1937045Z self_outputs = self.self( 2025-12-04T09:38:11.1937473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.1937950Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.1938493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.1939084Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.1939649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.1940154Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.1940325Z 2025-12-04T09:38:11.1940412Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.1940674Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1941247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1941770Z layer_outputs = layer_module( 2025-12-04T09:38:11.1942145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1942542Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1942988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1943441Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1943885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1944325Z self_outputs = self.self( 2025-12-04T09:38:11.1944744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.1945187Z attn_scores += diagonal_mask 2025-12-04T09:38:11.1945325Z 2025-12-04T09:38:11.1945455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1946009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1946524Z layer_outputs = layer_module( 2025-12-04T09:38:11.1946936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1947509Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1947960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1948420Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1948875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1949330Z self_outputs = self.self( 2025-12-04T09:38:11.1949830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.1950288Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.1950441Z 2025-12-04T09:38:11.1950554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1951110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1951625Z layer_outputs = layer_module( 2025-12-04T09:38:11.1951999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1952390Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1952842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1953288Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1953741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1954186Z self_outputs = self.self( 2025-12-04T09:38:11.1954608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:38:11.1955066Z value_vectors = self.value(hidden_states) 2025-12-04T09:38:11.1955223Z 2025-12-04T09:38:11.1955333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1955884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1956396Z layer_outputs = layer_module( 2025-12-04T09:38:11.1956776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1957174Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1957624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1958068Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1958520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1958966Z self_outputs = self.self( 2025-12-04T09:38:11.1959387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1959883Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1960448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1961166Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.1961634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.1962037Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.1962205Z 2025-12-04T09:38:11.1962382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1962949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1963474Z layer_outputs = layer_module( 2025-12-04T09:38:11.1963862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1964262Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1964722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1965210Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1965676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1966131Z self_outputs = self.self( 2025-12-04T09:38:11.1966567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1967079Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1967657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1968242Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.1968751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.1969231Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.1969582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.1969941Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.1970096Z 2025-12-04T09:38:11.1970206Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1970756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1971281Z layer_outputs = layer_module( 2025-12-04T09:38:11.1971655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1972045Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1972498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1972923Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1973337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1973757Z self_outputs = self.self( 2025-12-04T09:38:11.1974175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1974635Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1975156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1975723Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.1975935Z 2025-12-04T09:38:11.1976044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1976565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1977048Z layer_outputs = layer_module( 2025-12-04T09:38:11.1977454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1977823Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1978258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1978690Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1979124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1979549Z self_outputs = self.self( 2025-12-04T09:38:11.1979980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.1980446Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.1980985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.1981556Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.1981762Z 2025-12-04T09:38:11.1981868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1982389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1982878Z layer_outputs = layer_module( 2025-12-04T09:38:11.1983233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1983601Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1984032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1984454Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1984877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.1985301Z self_outputs = self.self( 2025-12-04T09:38:11.1985706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.1986242Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.1986486Z 2025-12-04T09:38:11.1986598Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1987125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1987643Z layer_outputs = layer_module( 2025-12-04T09:38:11.1988022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1988392Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1988818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.1989243Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.1989669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.1990125Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.1990589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.1991085Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.1991234Z 2025-12-04T09:38:11.1991353Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1991951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1992476Z layer_outputs = layer_module( 2025-12-04T09:38:11.1992851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1993242Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1993685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.1994148Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.1994628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.1995050Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.1995504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.1995994Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.1996486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.1996947Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.1997106Z 2025-12-04T09:38:11.1997220Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.1997791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.1998325Z layer_outputs = layer_module( 2025-12-04T09:38:11.1998713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.1999120Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.1999585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2000052Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2000582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2001035Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2001504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2002026Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2002518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.2003008Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.2003423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.2003790Z return self.act(input) 2025-12-04T09:38:11.2003923Z 2025-12-04T09:38:11.2004038Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2004592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2005117Z layer_outputs = layer_module( 2025-12-04T09:38:11.2005482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2005874Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2006324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2006770Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2007254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2007685Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2008130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.2008621Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.2009114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.2009566Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2009713Z 2025-12-04T09:38:11.2009867Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2010413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2010924Z layer_outputs = layer_module( 2025-12-04T09:38:11.2011281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2011643Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2012082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2012541Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2012965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2013376Z self_outputs = self.self( 2025-12-04T09:38:11.2013785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:38:11.2014218Z query_vectors = self.query(hidden_states) 2025-12-04T09:38:11.2014362Z 2025-12-04T09:38:11.2014480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2015024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2015545Z layer_outputs = layer_module( 2025-12-04T09:38:11.2015914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2016277Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2016704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2017133Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2017556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2017965Z self_outputs = self.self( 2025-12-04T09:38:11.2018373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2018825Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2019339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2019929Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2020182Z 2025-12-04T09:38:11.2020289Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2020814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2021309Z layer_outputs = layer_module( 2025-12-04T09:38:11.2021657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2022060Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2022485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2022908Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2023315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2023731Z self_outputs = self.self( 2025-12-04T09:38:11.2024130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:38:11.2024580Z key_vectors = self.key(hidden_states) 2025-12-04T09:38:11.2024732Z 2025-12-04T09:38:11.2024843Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2025390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2025906Z layer_outputs = layer_module( 2025-12-04T09:38:11.2026270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2026655Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2027075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2027520Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2027957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2028406Z self_outputs = self.self( 2025-12-04T09:38:11.2028831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2029294Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2029833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2030454Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2030711Z 2025-12-04T09:38:11.2030832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2031376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2031894Z layer_outputs = layer_module( 2025-12-04T09:38:11.2032267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2032660Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2033103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2033549Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2033994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2034436Z self_outputs = self.self( 2025-12-04T09:38:11.2034855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2035332Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2035867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2036485Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2036749Z 2025-12-04T09:38:11.2036861Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2037450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2037969Z layer_outputs = layer_module( 2025-12-04T09:38:11.2038336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2038727Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2039173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2039669Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2040111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2040643Z self_outputs = self.self( 2025-12-04T09:38:11.2041095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2041594Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2042123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2042773Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2043040Z 2025-12-04T09:38:11.2043138Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2043372Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2043611Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2043843Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2044103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2044670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2045204Z layer_outputs = layer_module( 2025-12-04T09:38:11.2045584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2045985Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2046443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2046900Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2047540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2048000Z self_outputs = self.self( 2025-12-04T09:38:11.2048443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.2048945Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2049506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2050107Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.2050696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.2051228Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.2051393Z 2025-12-04T09:38:11.2051488Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2051740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2052301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2052865Z layer_outputs = layer_module( 2025-12-04T09:38:11.2053222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2053595Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2054023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2054449Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2054740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2054857Z self_outputs = self.self( 2025-12-04T09:38:11.2055160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.2055235Z attn_scores += diagonal_mask 2025-12-04T09:38:11.2055239Z 2025-12-04T09:38:11.2055357Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2055725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2055799Z layer_outputs = layer_module( 2025-12-04T09:38:11.2056040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2056124Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2056422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2056503Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2056796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2056884Z self_outputs = self.self( 2025-12-04T09:38:11.2057174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.2057256Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.2057267Z 2025-12-04T09:38:11.2057374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2057741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2057819Z layer_outputs = layer_module( 2025-12-04T09:38:11.2058051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2058132Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2058430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2058506Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2058804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2058873Z self_outputs = self.self( 2025-12-04T09:38:11.2059162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:38:11.2059258Z value_vectors = self.value(hidden_states) 2025-12-04T09:38:11.2059261Z 2025-12-04T09:38:11.2059366Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2059739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2059813Z layer_outputs = layer_module( 2025-12-04T09:38:11.2060045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2060173Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2060460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2060537Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2060829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2060898Z self_outputs = self.self( 2025-12-04T09:38:11.2061189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2061345Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2061702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2061890Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.2062090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2062199Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2062202Z 2025-12-04T09:38:11.2062312Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2062673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2062755Z layer_outputs = layer_module( 2025-12-04T09:38:11.2062985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2063073Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2063361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2063442Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2063737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2063809Z self_outputs = self.self( 2025-12-04T09:38:11.2064094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2064224Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2064586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2064736Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.2065075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.2065168Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.2065370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2065471Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2065474Z 2025-12-04T09:38:11.2065588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2065942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2066018Z layer_outputs = layer_module( 2025-12-04T09:38:11.2066246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2066326Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2066644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2066722Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2066996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2067072Z self_outputs = self.self( 2025-12-04T09:38:11.2067347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2067461Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2067820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2068008Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2068012Z 2025-12-04T09:38:11.2068123Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2068483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2068553Z layer_outputs = layer_module( 2025-12-04T09:38:11.2068775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2068855Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2069135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2069213Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2069486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2069565Z self_outputs = self.self( 2025-12-04T09:38:11.2069840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2069961Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2070304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2070450Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2070454Z 2025-12-04T09:38:11.2070563Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2070912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2070990Z layer_outputs = layer_module( 2025-12-04T09:38:11.2071207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2071288Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2071571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2071645Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2071919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2071993Z self_outputs = self.self( 2025-12-04T09:38:11.2072265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.2072460Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.2072464Z 2025-12-04T09:38:11.2072563Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2072937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2073017Z layer_outputs = layer_module( 2025-12-04T09:38:11.2073237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2073320Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2073600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2073675Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2074003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.2074116Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.2074404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.2074488Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2074491Z 2025-12-04T09:38:11.2075017Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2075374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2075444Z layer_outputs = layer_module( 2025-12-04T09:38:11.2075754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2075861Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2076167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2076314Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2076642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2076758Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2077328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2077477Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2077815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.2077926Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2077932Z 2025-12-04T09:38:11.2078102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2078496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2078634Z layer_outputs = layer_module( 2025-12-04T09:38:11.2078937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2079044Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2079410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2079526Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2079817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2079996Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2080329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2080550Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2080964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.2081116Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.2081414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.2081533Z return self.act(input) 2025-12-04T09:38:11.2081538Z 2025-12-04T09:38:11.2102049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2102627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2102903Z layer_outputs = layer_module( 2025-12-04T09:38:11.2103153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2103253Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2103562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2103655Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2103935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2104018Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2104323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.2104456Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.2104762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.2104861Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2104870Z 2025-12-04T09:38:11.2104995Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2105398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2105484Z layer_outputs = layer_module( 2025-12-04T09:38:11.2105722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2105814Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2106109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2106202Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2106517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2106598Z self_outputs = self.self( 2025-12-04T09:38:11.2106918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:38:11.2107004Z query_vectors = self.query(hidden_states) 2025-12-04T09:38:11.2107009Z 2025-12-04T09:38:11.2107119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2107500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2107578Z layer_outputs = layer_module( 2025-12-04T09:38:11.2107835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2107926Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2108233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2108328Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2108726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2108804Z self_outputs = self.self( 2025-12-04T09:38:11.2109115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2109230Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2109612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2109858Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2109863Z 2025-12-04T09:38:11.2109980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2110362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2110447Z layer_outputs = layer_module( 2025-12-04T09:38:11.2110686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2110778Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2111079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2111162Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2111470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2111550Z self_outputs = self.self( 2025-12-04T09:38:11.2111856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:38:11.2111943Z key_vectors = self.key(hidden_states) 2025-12-04T09:38:11.2111947Z 2025-12-04T09:38:11.2112058Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2112443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2112518Z layer_outputs = layer_module( 2025-12-04T09:38:11.2112754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2112844Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2113146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2113232Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2113535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2113610Z self_outputs = self.self( 2025-12-04T09:38:11.2113916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2114028Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2114403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2114604Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2114611Z 2025-12-04T09:38:11.2114723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2115114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2115190Z layer_outputs = layer_module( 2025-12-04T09:38:11.2115481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2115569Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2115869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2115959Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2116258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2117289Z self_outputs = self.self( 2025-12-04T09:38:11.2117598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2117710Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2118087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2118289Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2118294Z 2025-12-04T09:38:11.2118404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2118791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2118868Z layer_outputs = layer_module( 2025-12-04T09:38:11.2119116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2119206Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2119506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2119600Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2119899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2119980Z self_outputs = self.self( 2025-12-04T09:38:11.2120276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2120384Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2120854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2121065Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2121070Z 2025-12-04T09:38:11.2121173Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2121263Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2121352Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2121447Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2121565Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2121964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2122053Z layer_outputs = layer_module( 2025-12-04T09:38:11.2122306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2122402Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2122708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2122789Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2123140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2123218Z self_outputs = self.self( 2025-12-04T09:38:11.2123518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.2123648Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2124016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2124184Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.2124576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.2124693Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.2124704Z 2025-12-04T09:38:11.2124786Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2124895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2125260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2125331Z layer_outputs = layer_module( 2025-12-04T09:38:11.2125571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2125666Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2125970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2126063Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2126365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2126439Z self_outputs = self.self( 2025-12-04T09:38:11.2126747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.2126828Z attn_scores += diagonal_mask 2025-12-04T09:38:11.2126831Z 2025-12-04T09:38:11.2126941Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2127327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2127402Z layer_outputs = layer_module( 2025-12-04T09:38:11.2127649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2127734Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2128037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2128126Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2128426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2128505Z self_outputs = self.self( 2025-12-04T09:38:11.2128802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.2128887Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.2128891Z 2025-12-04T09:38:11.2129008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2129386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2129467Z layer_outputs = layer_module( 2025-12-04T09:38:11.2129707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2129824Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2130134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2130213Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2130512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2130593Z self_outputs = self.self( 2025-12-04T09:38:11.2130888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:38:11.2131022Z value_vectors = self.value(hidden_states) 2025-12-04T09:38:11.2131026Z 2025-12-04T09:38:11.2131133Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2131515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2131598Z layer_outputs = layer_module( 2025-12-04T09:38:11.2131836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2131928Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2132233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2132312Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2132622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2132699Z self_outputs = self.self( 2025-12-04T09:38:11.2133001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2133141Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2133528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2133728Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.2133942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2134051Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2134062Z 2025-12-04T09:38:11.2134177Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2134560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2134644Z layer_outputs = layer_module( 2025-12-04T09:38:11.2134885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2134965Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2135263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2135339Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2135631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2135700Z self_outputs = self.self( 2025-12-04T09:38:11.2136000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2136135Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2136559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2136718Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.2137066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.2137164Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.2137388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2137494Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2137530Z 2025-12-04T09:38:11.2137642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2138026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2138105Z layer_outputs = layer_module( 2025-12-04T09:38:11.2138354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2138442Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2138744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2138835Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2139136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2139222Z self_outputs = self.self( 2025-12-04T09:38:11.2139522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2139648Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2140038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2140205Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2140210Z 2025-12-04T09:38:11.2140328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2140708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2140784Z layer_outputs = layer_module( 2025-12-04T09:38:11.2141030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2141117Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2141426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2141509Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2141809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2141890Z self_outputs = self.self( 2025-12-04T09:38:11.2142191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2142313Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2142697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2142862Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2142866Z 2025-12-04T09:38:11.2142983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2143396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2143474Z layer_outputs = layer_module( 2025-12-04T09:38:11.2143720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2143804Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2144113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2144194Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2144531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2144611Z self_outputs = self.self( 2025-12-04T09:38:11.2144951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.2145167Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.2145171Z 2025-12-04T09:38:11.2145283Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2145667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2145750Z layer_outputs = layer_module( 2025-12-04T09:38:11.2145992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2146078Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2146387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2146466Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2146779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.2146901Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.2147502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.2147611Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2147615Z 2025-12-04T09:38:11.2147723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2148113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2148193Z layer_outputs = layer_module( 2025-12-04T09:38:11.2148437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2148533Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2148839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2148939Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2149231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2149315Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2149637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2149764Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2150071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.2150168Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2150172Z 2025-12-04T09:38:11.2150374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2150762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2150837Z layer_outputs = layer_module( 2025-12-04T09:38:11.2151074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2151165Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2151463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2151619Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2151902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2151985Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2152302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2152423Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2152733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.2152856Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.2153092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.2153182Z return self.act(input) 2025-12-04T09:38:11.2153186Z 2025-12-04T09:38:11.2153296Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2153673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2153757Z layer_outputs = layer_module( 2025-12-04T09:38:11.2153981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2154068Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2154362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2154453Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2154742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2154827Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2155135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.2155260Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.2155550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.2155642Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2155645Z 2025-12-04T09:38:11.2155749Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2156169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2156246Z layer_outputs = layer_module( 2025-12-04T09:38:11.2156486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2156577Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2156876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2156992Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2157303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2157378Z self_outputs = self.self( 2025-12-04T09:38:11.2157680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:38:11.2157768Z query_vectors = self.query(hidden_states) 2025-12-04T09:38:11.2157772Z 2025-12-04T09:38:11.2157882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2158319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2158395Z layer_outputs = layer_module( 2025-12-04T09:38:11.2158645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2158734Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2159044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2159137Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2159448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2159530Z self_outputs = self.self( 2025-12-04T09:38:11.2159839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2159955Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2160340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2160614Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2160621Z 2025-12-04T09:38:11.2160738Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2161145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2161223Z layer_outputs = layer_module( 2025-12-04T09:38:11.2161478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2161569Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2161883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2161978Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2162268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2162346Z self_outputs = self.self( 2025-12-04T09:38:11.2162630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:38:11.2162713Z key_vectors = self.key(hidden_states) 2025-12-04T09:38:11.2162717Z 2025-12-04T09:38:11.2162827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2163183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2163266Z layer_outputs = layer_module( 2025-12-04T09:38:11.2163500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2163581Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2163907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2163997Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2164282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2164351Z self_outputs = self.self( 2025-12-04T09:38:11.2164643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2164747Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2165135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2165325Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2165328Z 2025-12-04T09:38:11.2165436Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2165803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2165875Z layer_outputs = layer_module( 2025-12-04T09:38:11.2166108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2166187Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2166468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2166554Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2166837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2166910Z self_outputs = self.self( 2025-12-04T09:38:11.2167194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2167295Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2167647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2167829Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2167833Z 2025-12-04T09:38:11.2167942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2168303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2168372Z layer_outputs = layer_module( 2025-12-04T09:38:11.2168604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2168684Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2168976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2169052Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2169336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2169411Z self_outputs = self.self( 2025-12-04T09:38:11.2169694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2169797Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2170146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2170360Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2170364Z 2025-12-04T09:38:11.2170455Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2170534Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2170611Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2170695Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2170798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2171157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2171274Z layer_outputs = layer_module( 2025-12-04T09:38:11.2171497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2171585Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2171873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2171947Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2172235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2172306Z self_outputs = self.self( 2025-12-04T09:38:11.2172593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.2172708Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2173051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2173208Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.2173536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.2173643Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.2173646Z 2025-12-04T09:38:11.2173724Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2173828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2174190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2174265Z layer_outputs = layer_module( 2025-12-04T09:38:11.2174491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2174579Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2174866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2174950Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2175240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2175313Z self_outputs = self.self( 2025-12-04T09:38:11.2175621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.2175699Z attn_scores += diagonal_mask 2025-12-04T09:38:11.2175703Z 2025-12-04T09:38:11.2175818Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2176195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2176270Z layer_outputs = layer_module( 2025-12-04T09:38:11.2176543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2176630Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2176942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2177021Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2177320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2177396Z self_outputs = self.self( 2025-12-04T09:38:11.2177688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.2177809Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.2177813Z 2025-12-04T09:38:11.2177929Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2178320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2178404Z layer_outputs = layer_module( 2025-12-04T09:38:11.2178648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2178732Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2179048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2179128Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2179447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2179520Z self_outputs = self.self( 2025-12-04T09:38:11.2179831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:38:11.2179928Z value_vectors = self.value(hidden_states) 2025-12-04T09:38:11.2179932Z 2025-12-04T09:38:11.2180040Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2180436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2180511Z layer_outputs = layer_module( 2025-12-04T09:38:11.2180753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2180847Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2181156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2181235Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2181556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2181630Z self_outputs = self.self( 2025-12-04T09:38:11.2181945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2182074Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2182467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2182669Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.2182888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2183003Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2183006Z 2025-12-04T09:38:11.2183146Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2183524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2183608Z layer_outputs = layer_module( 2025-12-04T09:38:11.2183846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2183937Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2184241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2184368Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2184674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2184749Z self_outputs = self.self( 2025-12-04T09:38:11.2185051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2185188Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2185570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2185727Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.2186071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.2186177Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.2186400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2186507Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2186511Z 2025-12-04T09:38:11.2186635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2187014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2187095Z layer_outputs = layer_module( 2025-12-04T09:38:11.2187341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2187429Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2187741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2187830Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2188133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2188218Z self_outputs = self.self( 2025-12-04T09:38:11.2188520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2188650Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2189040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2189209Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2189213Z 2025-12-04T09:38:11.2189334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2189719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2189800Z layer_outputs = layer_module( 2025-12-04T09:38:11.2190086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2190174Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2190481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2190561Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2190859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2190941Z self_outputs = self.self( 2025-12-04T09:38:11.2191239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2191411Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2191773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2191925Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2191929Z 2025-12-04T09:38:11.2192044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2192403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2192482Z layer_outputs = layer_module( 2025-12-04T09:38:11.2192707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2192790Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2193083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2193158Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2193443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2193521Z self_outputs = self.self( 2025-12-04T09:38:11.2193803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.2193998Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.2194002Z 2025-12-04T09:38:11.2194107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2194464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2194544Z layer_outputs = layer_module( 2025-12-04T09:38:11.2194769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2194858Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2195143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2195218Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2195510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.2195624Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.2195920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.2196007Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2196011Z 2025-12-04T09:38:11.2196115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2196530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2196608Z layer_outputs = layer_module( 2025-12-04T09:38:11.2196847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2196939Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2197243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2197341Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2197632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2197751Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2198064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2198189Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2198497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.2198587Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2198590Z 2025-12-04T09:38:11.2198701Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2199083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2199157Z layer_outputs = layer_module( 2025-12-04T09:38:11.2199405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2199489Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2199795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2199893Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2200173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2200253Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2200847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2200972Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2201290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.2201424Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.2201667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.2201756Z return self.act(input) 2025-12-04T09:38:11.2201775Z 2025-12-04T09:38:11.2201880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2202242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2202314Z layer_outputs = layer_module( 2025-12-04T09:38:11.2202537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2202626Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2202914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2203006Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2203271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2203390Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2203685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.2203810Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.2204096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.2204187Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2204191Z 2025-12-04T09:38:11.2204293Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2204694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2204765Z layer_outputs = layer_module( 2025-12-04T09:38:11.2204990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2205078Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2205365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2205449Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2205736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2205805Z self_outputs = self.self( 2025-12-04T09:38:11.2206087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:38:11.2206172Z query_vectors = self.query(hidden_states) 2025-12-04T09:38:11.2206175Z 2025-12-04T09:38:11.2206281Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2206629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2206698Z layer_outputs = layer_module( 2025-12-04T09:38:11.2206924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2207001Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2207276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2207362Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2207645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2207721Z self_outputs = self.self( 2025-12-04T09:38:11.2208000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2208101Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2208445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2208626Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2208629Z 2025-12-04T09:38:11.2208737Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2209084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2209157Z layer_outputs = layer_module( 2025-12-04T09:38:11.2209385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2209462Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2209785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2209863Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2210141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2210219Z self_outputs = self.self( 2025-12-04T09:38:11.2210493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:38:11.2210603Z key_vectors = self.key(hidden_states) 2025-12-04T09:38:11.2210606Z 2025-12-04T09:38:11.2210716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2211065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2211144Z layer_outputs = layer_module( 2025-12-04T09:38:11.2211363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2211440Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2211731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2211806Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2212093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2212167Z self_outputs = self.self( 2025-12-04T09:38:11.2212443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2212552Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2212898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2213086Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2213090Z 2025-12-04T09:38:11.2213192Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2213541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2213618Z layer_outputs = layer_module( 2025-12-04T09:38:11.2213843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2213920Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2214215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2214290Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2214628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2214698Z self_outputs = self.self( 2025-12-04T09:38:11.2214983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2215096Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2215445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2215641Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2215644Z 2025-12-04T09:38:11.2215748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2216162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2216243Z layer_outputs = layer_module( 2025-12-04T09:38:11.2216468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2216555Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2216841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2216951Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2217246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2217316Z self_outputs = self.self( 2025-12-04T09:38:11.2217610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2217719Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2218067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2218257Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2218261Z 2025-12-04T09:38:11.2218343Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2218423Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2218513Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2218591Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2218703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2219082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2219159Z layer_outputs = layer_module( 2025-12-04T09:38:11.2219406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2219498Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2219789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2219873Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2220165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2220249Z self_outputs = self.self( 2025-12-04T09:38:11.2220547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.2220666Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2221044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2221199Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.2221554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.2221659Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.2221663Z 2025-12-04T09:38:11.2221746Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2221867Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2222249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2222331Z layer_outputs = layer_module( 2025-12-04T09:38:11.2222611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2222699Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2223005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2223083Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2223372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2223489Z self_outputs = self.self( 2025-12-04T09:38:11.2223787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.2223873Z attn_scores += diagonal_mask 2025-12-04T09:38:11.2223877Z 2025-12-04T09:38:11.2223986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2224367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2224450Z layer_outputs = layer_module( 2025-12-04T09:38:11.2224690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2224781Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2225080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2225162Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2225466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2225540Z self_outputs = self.self( 2025-12-04T09:38:11.2225848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.2225936Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.2225939Z 2025-12-04T09:38:11.2226050Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2226431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2226519Z layer_outputs = layer_module( 2025-12-04T09:38:11.2226764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2226852Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2227157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2227235Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2227538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2227619Z self_outputs = self.self( 2025-12-04T09:38:11.2227916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:38:11.2228013Z value_vectors = self.value(hidden_states) 2025-12-04T09:38:11.2228016Z 2025-12-04T09:38:11.2228125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2228500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2228585Z layer_outputs = layer_module( 2025-12-04T09:38:11.2228821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2228912Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2229241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2229323Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2229632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2229706Z self_outputs = self.self( 2025-12-04T09:38:11.2230003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2230172Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2230551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2230749Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.2230960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2231065Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2231069Z 2025-12-04T09:38:11.2231188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2231563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2231646Z layer_outputs = layer_module( 2025-12-04T09:38:11.2231884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2231972Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2232278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2232359Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2232666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2232740Z self_outputs = self.self( 2025-12-04T09:38:11.2233035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2233168Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2233546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2233698Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.2234044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.2234143Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.2234356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2234459Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2234463Z 2025-12-04T09:38:11.2234572Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2234954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2235028Z layer_outputs = layer_module( 2025-12-04T09:38:11.2235274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2235357Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2235666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2235785Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2236067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2236145Z self_outputs = self.self( 2025-12-04T09:38:11.2236439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2236563Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2236943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2237141Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2237144Z 2025-12-04T09:38:11.2237263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2237645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2237721Z layer_outputs = layer_module( 2025-12-04T09:38:11.2237964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2238047Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2238347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2238438Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2238744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2238825Z self_outputs = self.self( 2025-12-04T09:38:11.2239130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2239252Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2239642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2239802Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2239805Z 2025-12-04T09:38:11.2239922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2240300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2240379Z layer_outputs = layer_module( 2025-12-04T09:38:11.2240716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2240815Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2241137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2241220Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2241529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2241616Z self_outputs = self.self( 2025-12-04T09:38:11.2241926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.2242147Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.2242158Z 2025-12-04T09:38:11.2242265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2242664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2242748Z layer_outputs = layer_module( 2025-12-04T09:38:11.2242978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2243060Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2243355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2243433Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2243729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.2243882Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.2244168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.2244271Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2244275Z 2025-12-04T09:38:11.2244379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2244745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2244816Z layer_outputs = layer_module( 2025-12-04T09:38:11.2245042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2245131Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2245422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2245515Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2245786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2245864Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2246162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2246273Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2246556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.2246648Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2246654Z 2025-12-04T09:38:11.2246760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2247304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2247381Z layer_outputs = layer_module( 2025-12-04T09:38:11.2247609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2247697Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2247981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2248075Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2248339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2248416Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2248714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2248826Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2249172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.2249298Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.2249518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.2249597Z return self.act(input) 2025-12-04T09:38:11.2249601Z 2025-12-04T09:38:11.2249707Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2250062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2250188Z layer_outputs = layer_module( 2025-12-04T09:38:11.2250412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2250498Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2250791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2250878Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2251152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2251229Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2251524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.2251650Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.2251938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.2252032Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2252036Z 2025-12-04T09:38:11.2252140Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2252500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2252579Z layer_outputs = layer_module( 2025-12-04T09:38:11.2252802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2252888Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2253171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2253253Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2253543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2253613Z self_outputs = self.self( 2025-12-04T09:38:11.2253905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:38:11.2253988Z query_vectors = self.query(hidden_states) 2025-12-04T09:38:11.2253992Z 2025-12-04T09:38:11.2254094Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2254453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2254523Z layer_outputs = layer_module( 2025-12-04T09:38:11.2254754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2254835Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2255128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2255219Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2255550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2255637Z self_outputs = self.self( 2025-12-04T09:38:11.2255925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2256029Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2256386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2256615Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2256619Z 2025-12-04T09:38:11.2256722Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2257086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2257158Z layer_outputs = layer_module( 2025-12-04T09:38:11.2257389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2257471Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2257772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2257861Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2258161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2258245Z self_outputs = self.self( 2025-12-04T09:38:11.2258543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:38:11.2258628Z key_vectors = self.key(hidden_states) 2025-12-04T09:38:11.2258634Z 2025-12-04T09:38:11.2258752Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2259127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2259209Z layer_outputs = layer_module( 2025-12-04T09:38:11.2259446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2259530Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2259838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2259921Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2260221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2260305Z self_outputs = self.self( 2025-12-04T09:38:11.2260606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2260720Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2261084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2261280Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2261284Z 2025-12-04T09:38:11.2261404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2261776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2261856Z layer_outputs = layer_module( 2025-12-04T09:38:11.2262125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2262211Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2262526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2262604Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2262920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2262993Z self_outputs = self.self( 2025-12-04T09:38:11.2263331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2263448Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2263818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2264013Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2264023Z 2025-12-04T09:38:11.2264134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2264513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2264595Z layer_outputs = layer_module( 2025-12-04T09:38:11.2264844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2264928Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2265238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2265318Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2265633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2265706Z self_outputs = self.self( 2025-12-04T09:38:11.2266006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2266123Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2266493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2266697Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2266700Z 2025-12-04T09:38:11.2266788Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2266873Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2266966Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2267051Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2267159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2267550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2267625Z layer_outputs = layer_module( 2025-12-04T09:38:11.2267868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2267953Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2268259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2268347Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2268691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2268774Z self_outputs = self.self( 2025-12-04T09:38:11.2269072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.2269189Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2269556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2269710Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.2270095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.2270208Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.2270212Z 2025-12-04T09:38:11.2270295Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2270413Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2270798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2270874Z layer_outputs = layer_module( 2025-12-04T09:38:11.2271123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2271208Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2271527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2271612Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2271936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2272017Z self_outputs = self.self( 2025-12-04T09:38:11.2272324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.2272402Z attn_scores += diagonal_mask 2025-12-04T09:38:11.2272414Z 2025-12-04T09:38:11.2272521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2272915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2272998Z layer_outputs = layer_module( 2025-12-04T09:38:11.2273258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2273343Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2273658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2273739Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2274058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2274134Z self_outputs = self.self( 2025-12-04T09:38:11.2274440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.2274533Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.2274536Z 2025-12-04T09:38:11.2274647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2275035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2275114Z layer_outputs = layer_module( 2025-12-04T09:38:11.2275358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2275485Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2275794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2275874Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2276183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2276256Z self_outputs = self.self( 2025-12-04T09:38:11.2276565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:38:11.2276696Z value_vectors = self.value(hidden_states) 2025-12-04T09:38:11.2276700Z 2025-12-04T09:38:11.2276811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2277197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2277273Z layer_outputs = layer_module( 2025-12-04T09:38:11.2277517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2277601Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2277904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2277993Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2278299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2278383Z self_outputs = self.self( 2025-12-04T09:38:11.2278682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2278813Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2279198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2279386Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.2279598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2279714Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2279718Z 2025-12-04T09:38:11.2279827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2280223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2280298Z layer_outputs = layer_module( 2025-12-04T09:38:11.2280640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2280739Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2281051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2281146Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2281468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2281544Z self_outputs = self.self( 2025-12-04T09:38:11.2281868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2281997Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2282427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2282578Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.2282920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.2283035Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.2283246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2283353Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2283365Z 2025-12-04T09:38:11.2283522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2283904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2283988Z layer_outputs = layer_module( 2025-12-04T09:38:11.2284228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2284311Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2284618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2284692Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2284984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2285054Z self_outputs = self.self( 2025-12-04T09:38:11.2285339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2285461Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2285822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2285984Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2285987Z 2025-12-04T09:38:11.2286089Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2286449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2286527Z layer_outputs = layer_module( 2025-12-04T09:38:11.2286753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2286842Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2287125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2287200Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2287491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2287561Z self_outputs = self.self( 2025-12-04T09:38:11.2287842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2287964Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2288316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2288475Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2288479Z 2025-12-04T09:38:11.2288583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2288974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2289053Z layer_outputs = layer_module( 2025-12-04T09:38:11.2289277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2289362Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2289643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2289717Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2290003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2290108Z self_outputs = self.self( 2025-12-04T09:38:11.2290396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.2290588Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.2290592Z 2025-12-04T09:38:11.2290696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2291064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2291137Z layer_outputs = layer_module( 2025-12-04T09:38:11.2291378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2291464Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2291768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2291850Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2292134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.2292247Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.2292537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.2292623Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2292626Z 2025-12-04T09:38:11.2292734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2293088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2293161Z layer_outputs = layer_module( 2025-12-04T09:38:11.2293394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2293473Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2293766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2293852Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2294119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2294206Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2294495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2294619Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2294918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.2295007Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2295011Z 2025-12-04T09:38:11.2295161Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2295538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2295614Z layer_outputs = layer_module( 2025-12-04T09:38:11.2295861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2295943Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2296250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2296376Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2296660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2296749Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2297056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2297180Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2297482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.2297606Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.2297846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.2297925Z return self.act(input) 2025-12-04T09:38:11.2297929Z 2025-12-04T09:38:11.2298046Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2298409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2298490Z layer_outputs = layer_module( 2025-12-04T09:38:11.2298721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2298801Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2299086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2299179Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2299460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2299552Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2299856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.2299987Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.2300300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.2300389Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2300392Z 2025-12-04T09:38:11.2300510Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2300884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2300959Z layer_outputs = layer_module( 2025-12-04T09:38:11.2301202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2301287Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2301586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2301676Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2302013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2302100Z self_outputs = self.self( 2025-12-04T09:38:11.2302401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:38:11.2302488Z query_vectors = self.query(hidden_states) 2025-12-04T09:38:11.2302492Z 2025-12-04T09:38:11.2302617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2302974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2303089Z layer_outputs = layer_module( 2025-12-04T09:38:11.2303322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2303404Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2303709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2303789Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2304107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2304180Z self_outputs = self.self( 2025-12-04T09:38:11.2304491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2304609Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2304987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2305190Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2305201Z 2025-12-04T09:38:11.2305312Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2305701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2305784Z layer_outputs = layer_module( 2025-12-04T09:38:11.2306026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2306109Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2306426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2306507Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2306829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2306903Z self_outputs = self.self( 2025-12-04T09:38:11.2307211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:38:11.2307302Z key_vectors = self.key(hidden_states) 2025-12-04T09:38:11.2307306Z 2025-12-04T09:38:11.2307416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2307812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2307890Z layer_outputs = layer_module( 2025-12-04T09:38:11.2308133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2308224Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2308579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2308661Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2308975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2309048Z self_outputs = self.self( 2025-12-04T09:38:11.2309361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2309470Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2309876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2310080Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2310083Z 2025-12-04T09:38:11.2310193Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2310579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2310656Z layer_outputs = layer_module( 2025-12-04T09:38:11.2310893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2310984Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2311287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2311379Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2311681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2311756Z self_outputs = self.self( 2025-12-04T09:38:11.2312069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2312176Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2312552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2312746Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2312750Z 2025-12-04T09:38:11.2312859Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2313257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2313328Z layer_outputs = layer_module( 2025-12-04T09:38:11.2313561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2313641Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2313929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2314012Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2314298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2314369Z self_outputs = self.self( 2025-12-04T09:38:11.2314663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2314766Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2315123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2315336Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2315340Z 2025-12-04T09:38:11.2315424Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2315515Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2315597Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2315684Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2315788Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2316161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2316311Z layer_outputs = layer_module( 2025-12-04T09:38:11.2316548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2316634Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2316950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2317030Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2317344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2317415Z self_outputs = self.self( 2025-12-04T09:38:11.2317697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.2317821Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2318190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2318350Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.2318702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.2318805Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.2318809Z 2025-12-04T09:38:11.2318899Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2319008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2319386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2319470Z layer_outputs = layer_module( 2025-12-04T09:38:11.2319716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2319806Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2320109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2320192Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2320572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2320654Z self_outputs = self.self( 2025-12-04T09:38:11.2320962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.2321044Z attn_scores += diagonal_mask 2025-12-04T09:38:11.2321048Z 2025-12-04T09:38:11.2321163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2321564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2321642Z layer_outputs = layer_module( 2025-12-04T09:38:11.2321937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2322024Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2322324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2322412Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2322712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2322786Z self_outputs = self.self( 2025-12-04T09:38:11.2323093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.2323216Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.2323219Z 2025-12-04T09:38:11.2323336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2323716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2323790Z layer_outputs = layer_module( 2025-12-04T09:38:11.2324037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2324120Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2324431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2324510Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2324814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2324896Z self_outputs = self.self( 2025-12-04T09:38:11.2325198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:38:11.2325290Z value_vectors = self.value(hidden_states) 2025-12-04T09:38:11.2325300Z 2025-12-04T09:38:11.2325409Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2325788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2325868Z layer_outputs = layer_module( 2025-12-04T09:38:11.2326111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2326193Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2326503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2326582Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2326891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2326964Z self_outputs = self.self( 2025-12-04T09:38:11.2327264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2327398Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2327777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2327970Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.2328185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2328289Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2328293Z 2025-12-04T09:38:11.2328407Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2328856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2328940Z layer_outputs = layer_module( 2025-12-04T09:38:11.2329184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2329267Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2329578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2329695Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2330003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2330086Z self_outputs = self.self( 2025-12-04T09:38:11.2330385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2330518Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2330897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2331043Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.2331393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.2331492Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.2331710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2331816Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2331819Z 2025-12-04T09:38:11.2331932Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2332324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2332401Z layer_outputs = layer_module( 2025-12-04T09:38:11.2332647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2332730Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2333032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2333124Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2333426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2333500Z self_outputs = self.self( 2025-12-04T09:38:11.2333808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2333932Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2334316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2334480Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2334483Z 2025-12-04T09:38:11.2334593Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2334997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2335075Z layer_outputs = layer_module( 2025-12-04T09:38:11.2335382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2335505Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2335810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2335900Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2336204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2336285Z self_outputs = self.self( 2025-12-04T09:38:11.2336585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2336744Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2337129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2337294Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2337297Z 2025-12-04T09:38:11.2337414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2337795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2337872Z layer_outputs = layer_module( 2025-12-04T09:38:11.2338117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2338204Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2338527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2338616Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2338939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2339022Z self_outputs = self.self( 2025-12-04T09:38:11.2339335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.2339540Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.2339544Z 2025-12-04T09:38:11.2339667Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2340066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2340153Z layer_outputs = layer_module( 2025-12-04T09:38:11.2340391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2340476Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2340785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2340864Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2341171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.2341290Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.2341592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.2341693Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2341696Z 2025-12-04T09:38:11.2341803Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2342208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2342292Z layer_outputs = layer_module( 2025-12-04T09:38:11.2342536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2342631Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2342940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2343034Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2343336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2343455Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2343785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2343905Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2344211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.2344308Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2344312Z 2025-12-04T09:38:11.2344422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2344807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2344884Z layer_outputs = layer_module( 2025-12-04T09:38:11.2345131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2345223Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2345546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2345643Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2345944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2346028Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2346349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2346471Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2346791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.2346928Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.2347339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.2347432Z return self.act(input) 2025-12-04T09:38:11.2347436Z 2025-12-04T09:38:11.2347555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2347948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2348037Z layer_outputs = layer_module( 2025-12-04T09:38:11.2348281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2348376Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2348696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2348792Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2349092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2349239Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2349553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.2349698Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.2350014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.2350112Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2350116Z 2025-12-04T09:38:11.2350230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2350669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2350757Z layer_outputs = layer_module( 2025-12-04T09:38:11.2351005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2351102Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2351411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2351496Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2351815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2351892Z self_outputs = self.self( 2025-12-04T09:38:11.2352202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:38:11.2352305Z query_vectors = self.query(hidden_states) 2025-12-04T09:38:11.2352309Z 2025-12-04T09:38:11.2352422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2352819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2352898Z layer_outputs = layer_module( 2025-12-04T09:38:11.2353141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2353236Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2353545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2353645Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2353944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2354021Z self_outputs = self.self( 2025-12-04T09:38:11.2354326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2354436Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2354808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2355005Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2355008Z 2025-12-04T09:38:11.2355116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2355500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2355577Z layer_outputs = layer_module( 2025-12-04T09:38:11.2355819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2355903Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2356235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2356326Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2356633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2356706Z self_outputs = self.self( 2025-12-04T09:38:11.2357020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:38:11.2357104Z key_vectors = self.key(hidden_states) 2025-12-04T09:38:11.2357150Z 2025-12-04T09:38:11.2357270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2357642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2357717Z layer_outputs = layer_module( 2025-12-04T09:38:11.2357968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2358051Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2358368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2358449Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2358757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2358845Z self_outputs = self.self( 2025-12-04T09:38:11.2359155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2359268Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2359657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2359858Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2359862Z 2025-12-04T09:38:11.2359982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2360367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2360500Z layer_outputs = layer_module( 2025-12-04T09:38:11.2360765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2360851Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2361170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2361253Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2361566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2361650Z self_outputs = self.self( 2025-12-04T09:38:11.2361961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2362076Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2362460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2362656Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2362660Z 2025-12-04T09:38:11.2362775Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2363193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2363276Z layer_outputs = layer_module( 2025-12-04T09:38:11.2363516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2363600Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2363909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2363990Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2364335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2364410Z self_outputs = self.self( 2025-12-04T09:38:11.2364712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2364828Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2365196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2365387Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2365398Z 2025-12-04T09:38:11.2365485Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2365569Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2365658Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2365742Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2365851Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2366237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2366315Z layer_outputs = layer_module( 2025-12-04T09:38:11.2366556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2366646Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2366949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2367036Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2367335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2367412Z self_outputs = self.self( 2025-12-04T09:38:11.2367721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.2367838Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2368213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2368367Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.2368717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.2368828Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.2368832Z 2025-12-04T09:38:11.2368916Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2369034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2369413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2369490Z layer_outputs = layer_module( 2025-12-04T09:38:11.2369774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2369863Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2370167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2370255Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2370556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2370643Z self_outputs = self.self( 2025-12-04T09:38:11.2370958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.2371032Z attn_scores += diagonal_mask 2025-12-04T09:38:11.2371035Z 2025-12-04T09:38:11.2371147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2371520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2371609Z layer_outputs = layer_module( 2025-12-04T09:38:11.2371843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2371925Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2372231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2372312Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2372633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2372702Z self_outputs = self.self( 2025-12-04T09:38:11.2372988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.2373077Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.2373080Z 2025-12-04T09:38:11.2373182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2373533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2373613Z layer_outputs = layer_module( 2025-12-04T09:38:11.2373835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2373924Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2374208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2374282Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2374587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2374660Z self_outputs = self.self( 2025-12-04T09:38:11.2374964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:38:11.2375054Z value_vectors = self.value(hidden_states) 2025-12-04T09:38:11.2375057Z 2025-12-04T09:38:11.2375167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2375548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2375625Z layer_outputs = layer_module( 2025-12-04T09:38:11.2375870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2375955Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2376307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2376391Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2376676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2376746Z self_outputs = self.self( 2025-12-04T09:38:11.2377039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2377159Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2377624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2377808Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.2378019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2378133Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2378137Z 2025-12-04T09:38:11.2378246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2378625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2378700Z layer_outputs = layer_module( 2025-12-04T09:38:11.2378937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2379031Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2379338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2379419Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2379702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2379773Z self_outputs = self.self( 2025-12-04T09:38:11.2380063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2380181Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2380537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2380695Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.2381039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.2381149Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.2381357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2381462Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2381466Z 2025-12-04T09:38:11.2381583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2381960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2382044Z layer_outputs = layer_module( 2025-12-04T09:38:11.2382281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2382368Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2382674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2382788Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2383106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2383177Z self_outputs = self.self( 2025-12-04T09:38:11.2383461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2383585Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2383943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2384145Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2384155Z 2025-12-04T09:38:11.2384262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2384645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2384728Z layer_outputs = layer_module( 2025-12-04T09:38:11.2384976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2385056Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2385349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2385426Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2385717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2385787Z self_outputs = self.self( 2025-12-04T09:38:11.2386070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2386195Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2386552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2386711Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2386715Z 2025-12-04T09:38:11.2386821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2387175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2387257Z layer_outputs = layer_module( 2025-12-04T09:38:11.2387482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2387568Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2387853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2387931Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2388241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2388313Z self_outputs = self.self( 2025-12-04T09:38:11.2388617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.2388818Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.2388821Z 2025-12-04T09:38:11.2388923Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2389315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2389389Z layer_outputs = layer_module( 2025-12-04T09:38:11.2389616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2389705Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2390013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2390098Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2390407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.2390561Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.2390867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.2390959Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2390963Z 2025-12-04T09:38:11.2391079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2391452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2391527Z layer_outputs = layer_module( 2025-12-04T09:38:11.2391772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2391855Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2392157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2392256Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2392542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2392631Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2392936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2393054Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2393360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.2393451Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2393455Z 2025-12-04T09:38:11.2393572Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2393950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2394024Z layer_outputs = layer_module( 2025-12-04T09:38:11.2394269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2394353Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2394660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2394750Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2395029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2395118Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2395423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2395538Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2395876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.2396001Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.2396240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.2396316Z return self.act(input) 2025-12-04T09:38:11.2396320Z 2025-12-04T09:38:11.2396431Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2396822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2396935Z layer_outputs = layer_module( 2025-12-04T09:38:11.2397189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2397274Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2397596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2397694Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2397994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2398082Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2398404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.2398540Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.2398863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.2398954Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2398958Z 2025-12-04T09:38:11.2399071Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2399480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2399559Z layer_outputs = layer_module( 2025-12-04T09:38:11.2399814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2399900Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2400217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2400309Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2400688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2400779Z self_outputs = self.self( 2025-12-04T09:38:11.2401096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:38:11.2401187Z query_vectors = self.query(hidden_states) 2025-12-04T09:38:11.2401191Z 2025-12-04T09:38:11.2401314Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2401701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2401787Z layer_outputs = layer_module( 2025-12-04T09:38:11.2402033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2402123Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2402446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2402530Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2402873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2402960Z self_outputs = self.self( 2025-12-04T09:38:11.2403261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2403379Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2403747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2403946Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2403987Z 2025-12-04T09:38:11.2404110Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2404487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2404573Z layer_outputs = layer_module( 2025-12-04T09:38:11.2404813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2404896Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2405203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2405284Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2405592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2405669Z self_outputs = self.self( 2025-12-04T09:38:11.2405970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:38:11.2406060Z key_vectors = self.key(hidden_states) 2025-12-04T09:38:11.2406064Z 2025-12-04T09:38:11.2406174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2406560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2406642Z layer_outputs = layer_module( 2025-12-04T09:38:11.2406877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2406966Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2407269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2407352Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2407658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2407731Z self_outputs = self.self( 2025-12-04T09:38:11.2408038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2408148Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2408513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2408714Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2408718Z 2025-12-04T09:38:11.2408826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2409209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2409279Z layer_outputs = layer_module( 2025-12-04T09:38:11.2409546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2409637Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2409920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2409997Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2410290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2410359Z self_outputs = self.self( 2025-12-04T09:38:11.2410649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2410785Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2411133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2411323Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2411326Z 2025-12-04T09:38:11.2411431Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2411814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2411889Z layer_outputs = layer_module( 2025-12-04T09:38:11.2412125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2412220Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2412521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2412608Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2412910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2412982Z self_outputs = self.self( 2025-12-04T09:38:11.2413288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2413396Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2413767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2413962Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2413965Z 2025-12-04T09:38:11.2414053Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2414145Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2414226Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2414310Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2414429Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2414802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2414884Z layer_outputs = layer_module( 2025-12-04T09:38:11.2415121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2415204Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2415512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2415596Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2415900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2416006Z self_outputs = self.self( 2025-12-04T09:38:11.2416291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.2416410Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2416756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2416902Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.2417240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.2417370Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.2417374Z 2025-12-04T09:38:11.2417460Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2417563Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2417925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2418004Z layer_outputs = layer_module( 2025-12-04T09:38:11.2418234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2418323Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2418631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2418716Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2419028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2419102Z self_outputs = self.self( 2025-12-04T09:38:11.2419407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.2419492Z attn_scores += diagonal_mask 2025-12-04T09:38:11.2419495Z 2025-12-04T09:38:11.2419601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2420051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2420122Z layer_outputs = layer_module( 2025-12-04T09:38:11.2420347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2420435Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2420724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2420809Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2421101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2421171Z self_outputs = self.self( 2025-12-04T09:38:11.2421463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.2421543Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.2421546Z 2025-12-04T09:38:11.2421655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2422025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2422103Z layer_outputs = layer_module( 2025-12-04T09:38:11.2422349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2422432Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2422770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2422865Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2423166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2423248Z self_outputs = self.self( 2025-12-04T09:38:11.2423547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:38:11.2423701Z value_vectors = self.value(hidden_states) 2025-12-04T09:38:11.2423705Z 2025-12-04T09:38:11.2423821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2424203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2424288Z layer_outputs = layer_module( 2025-12-04T09:38:11.2424529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2424613Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2424922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2425002Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2425311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2425385Z self_outputs = self.self( 2025-12-04T09:38:11.2425678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2425812Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2426224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2426411Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.2426632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2426738Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2426742Z 2025-12-04T09:38:11.2426858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2427244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2427318Z layer_outputs = layer_module( 2025-12-04T09:38:11.2427570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2427654Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2427964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2428043Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2428346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2428427Z self_outputs = self.self( 2025-12-04T09:38:11.2428729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2428866Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2429246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2429426Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.2429785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.2429884Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.2430093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2430206Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2430209Z 2025-12-04T09:38:11.2430320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2430735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2430811Z layer_outputs = layer_module( 2025-12-04T09:38:11.2431050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2431142Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2431450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2431538Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2431849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2431924Z self_outputs = self.self( 2025-12-04T09:38:11.2432240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2432370Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2432767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2432936Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2432940Z 2025-12-04T09:38:11.2433051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2433446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2433524Z layer_outputs = layer_module( 2025-12-04T09:38:11.2433775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2433866Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2434177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2434264Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2434566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2434640Z self_outputs = self.self( 2025-12-04T09:38:11.2434948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2435070Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2435452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2435616Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2435620Z 2025-12-04T09:38:11.2435729Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2436157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2436234Z layer_outputs = layer_module( 2025-12-04T09:38:11.2436483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2436566Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2436868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2436954Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2437263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2437380Z self_outputs = self.self( 2025-12-04T09:38:11.2437688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.2437896Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.2437900Z 2025-12-04T09:38:11.2438019Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2438407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2438494Z layer_outputs = layer_module( 2025-12-04T09:38:11.2438738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2438823Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2439142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2439224Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2439533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.2439665Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.2439975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.2440075Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2440079Z 2025-12-04T09:38:11.2440190Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2440655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2440753Z layer_outputs = layer_module( 2025-12-04T09:38:11.2441002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2441096Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2441414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2441508Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2441809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2441894Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2442208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2442344Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2442658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.2442759Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2442763Z 2025-12-04T09:38:11.2442875Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2443302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2443390Z layer_outputs = layer_module( 2025-12-04T09:38:11.2443639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2443735Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2444053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2444184Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2444492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2444575Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2444897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2445019Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2445331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.2445467Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.2445703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.2445783Z return self.act(input) 2025-12-04T09:38:11.2445797Z 2025-12-04T09:38:11.2445910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2446304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2446392Z layer_outputs = layer_module( 2025-12-04T09:38:11.2446638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2446724Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2447242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2447346Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2447660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2447742Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2448070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.2448217Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.2448542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.2448641Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2448645Z 2025-12-04T09:38:11.2448758Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2449146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2449236Z layer_outputs = layer_module( 2025-12-04T09:38:11.2449483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2449575Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2449896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2449980Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2450375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2450454Z self_outputs = self.self( 2025-12-04T09:38:11.2450767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:38:11.2450866Z query_vectors = self.query(hidden_states) 2025-12-04T09:38:11.2450870Z 2025-12-04T09:38:11.2450981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2451389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2451512Z layer_outputs = layer_module( 2025-12-04T09:38:11.2451761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2451856Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2452188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2452281Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2452600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2452677Z self_outputs = self.self( 2025-12-04T09:38:11.2453008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2453128Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2453518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2453742Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2453748Z 2025-12-04T09:38:11.2453854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2454228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2454299Z layer_outputs = layer_module( 2025-12-04T09:38:11.2454542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2454633Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2454947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2455033Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2455336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2455408Z self_outputs = self.self( 2025-12-04T09:38:11.2455707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:38:11.2455787Z key_vectors = self.key(hidden_states) 2025-12-04T09:38:11.2455790Z 2025-12-04T09:38:11.2455901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2456270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2456342Z layer_outputs = layer_module( 2025-12-04T09:38:11.2456584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2456663Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2456991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2457078Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2457361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2457439Z self_outputs = self.self( 2025-12-04T09:38:11.2457720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2457823Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2458181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2458410Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2458413Z 2025-12-04T09:38:11.2458526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2458890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2458960Z layer_outputs = layer_module( 2025-12-04T09:38:11.2459193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2459272Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2459564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2459641Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2459927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2460007Z self_outputs = self.self( 2025-12-04T09:38:11.2460293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2460395Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2460748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2460933Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2460937Z 2025-12-04T09:38:11.2461049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2461401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2461484Z layer_outputs = layer_module( 2025-12-04T09:38:11.2461711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2461793Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2462087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2462164Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2462447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2462526Z self_outputs = self.self( 2025-12-04T09:38:11.2462810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:38:11.2462921Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2463264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2463475Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:38:11.2463479Z 2025-12-04T09:38:11.2463573Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2463656Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2463740Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2463817Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2463919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2464285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2464392Z layer_outputs = layer_module( 2025-12-04T09:38:11.2464619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2464707Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2464997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2465082Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2465368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2465437Z self_outputs = self.self( 2025-12-04T09:38:11.2465724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:38:11.2465845Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:38:11.2466193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:38:11.2466333Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:38:11.2466655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:38:11.2466760Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:38:11.2466763Z 2025-12-04T09:38:11.2466841Z cudagraph partition due to non gpu ops 2025-12-04T09:38:11.2466941Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2467293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2467364Z layer_outputs = layer_module( 2025-12-04T09:38:11.2467598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2467680Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2467963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2468050Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2468335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2468413Z self_outputs = self.self( 2025-12-04T09:38:11.2468698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:38:11.2468773Z attn_scores += diagonal_mask 2025-12-04T09:38:11.2468776Z 2025-12-04T09:38:11.2468891Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2469271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2469355Z layer_outputs = layer_module( 2025-12-04T09:38:11.2469598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2469704Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2470001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2470077Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2470364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2470441Z self_outputs = self.self( 2025-12-04T09:38:11.2470727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:38:11.2470856Z attn_probs = nn.functional.softmax( 2025-12-04T09:38:11.2470859Z 2025-12-04T09:38:11.2470971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2471351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2471439Z layer_outputs = layer_module( 2025-12-04T09:38:11.2471677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2471767Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2472068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2472147Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2472451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2472528Z self_outputs = self.self( 2025-12-04T09:38:11.2472827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:38:11.2472920Z value_vectors = self.value(hidden_states) 2025-12-04T09:38:11.2472923Z 2025-12-04T09:38:11.2473029Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2473395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2473465Z layer_outputs = layer_module( 2025-12-04T09:38:11.2473693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2473782Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2474070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2474152Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2474437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2474509Z self_outputs = self.self( 2025-12-04T09:38:11.2474801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2474922Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2475288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2475466Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:38:11.2475668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2475777Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2475781Z 2025-12-04T09:38:11.2475889Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2476314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2476393Z layer_outputs = layer_module( 2025-12-04T09:38:11.2476630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2476721Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2477023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2477103Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2477449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2477522Z self_outputs = self.self( 2025-12-04T09:38:11.2477830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2477956Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2478339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2478495Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:38:11.2478839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:38:11.2478946Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:38:11.2479159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:38:11.2479263Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:38:11.2479267Z 2025-12-04T09:38:11.2479381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2479764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2479845Z layer_outputs = layer_module( 2025-12-04T09:38:11.2480085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2480168Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2480554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2480651Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2480972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2481057Z self_outputs = self.self( 2025-12-04T09:38:11.2481369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2481504Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2481902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2482059Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2482063Z 2025-12-04T09:38:11.2482177Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2482535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2482620Z layer_outputs = layer_module( 2025-12-04T09:38:11.2482845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2482972Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2483267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2483344Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2483632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2483703Z self_outputs = self.self( 2025-12-04T09:38:11.2483983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:38:11.2484154Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:38:11.2484512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:38:11.2484664Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:38:11.2484676Z 2025-12-04T09:38:11.2484778Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2485135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2485213Z layer_outputs = layer_module( 2025-12-04T09:38:11.2485434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2485512Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2485806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2485881Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2486174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:38:11.2486243Z self_outputs = self.self( 2025-12-04T09:38:11.2486523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:38:11.2486716Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:38:11.2486720Z 2025-12-04T09:38:11.2486822Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2487185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2487259Z layer_outputs = layer_module( 2025-12-04T09:38:11.2487481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2487566Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2487860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:38:11.2487939Z self_attn_outputs = self.attention( 2025-12-04T09:38:11.2488213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:38:11.2488324Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:38:11.2488609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:38:11.2488694Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2488697Z 2025-12-04T09:38:11.2488795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2489150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2489247Z layer_outputs = layer_module( 2025-12-04T09:38:11.2489472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2489547Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2489822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2489912Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2490169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2490293Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2490574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2490683Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2490968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:38:11.2491049Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2491054Z 2025-12-04T09:38:11.2491159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2491506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2491576Z layer_outputs = layer_module( 2025-12-04T09:38:11.2491799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2491877Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2492153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2492244Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2492500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2492581Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2492862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:38:11.2492970Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:38:11.2493250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:38:11.2493366Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:38:11.2493585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:11.2493655Z return self.act(input) 2025-12-04T09:38:11.2493659Z 2025-12-04T09:38:11.2493760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:11.2494119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:38:11.2494192Z layer_outputs = layer_module( 2025-12-04T09:38:11.2494416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:11.2494509Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:11.2494787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:38:11.2494879Z layer_output = apply_chunking_to_forward( 2025-12-04T09:38:11.2495134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:11.2495209Z return forward_fn(*input_tensors) 2025-12-04T09:38:11.2495521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:38:11.2495646Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:38:11.2495934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:38:11.2496014Z hidden_states = self.dense(hidden_states) 2025-12-04T09:38:11.2496017Z 2025-12-04T09:39:21.2784349Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:21.2788700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1715, in torch_dynamo_resume_in_forward_at_1702 2025-12-04T09:39:21.2789759Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T09:39:21.2790364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1332, in forward 2025-12-04T09:39:21.2790848Z x = self.dense(features) 2025-12-04T09:39:21.2790993Z 2025-12-04T09:39:21.2791119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:21.2791705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1715, in torch_dynamo_resume_in_forward_at_1702 2025-12-04T09:39:21.2792274Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T09:39:21.2792799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1337, in forward 2025-12-04T09:39:21.2793385Z x = self.decoder(x) 2025-12-04T09:39:21.2793492Z 2025-12-04T09:39:21.2793604Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:21.2794118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1722, in torch_dynamo_resume_in_forward_at_1702 2025-12-04T09:39:21.2794721Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:39:21.2794968Z 2025-12-04T09:39:22.7084033Z Compilation time (from dynamo_timed): 103.528173962 2025-12-04T09:39:22.7383525Z pass 2025-12-04T09:39:22.7383980Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:22.7384854Z TIMING: gc:0.00678 entire_frame_compile:103.52817 _recursive_pre_grad_passes:0.01845 _recursive_joint_graph_passes:0.97048 _recursive_post_grad_passes:1.56213 async_compile.wait:3.18564 code_gen:80.40364 inductor_compile:87.87822 backend_compile:97.92307 total_wall_time:103.52817 2025-12-04T09:39:22.7386043Z STATS: call_* op count: 1787 | FakeTensorMode.__torch_dispatch__:40001 | FakeTensor.__torch_dispatch__:16210 | ProxyTorchDispatchMode.__torch_dispatch__:10021 2025-12-04T09:39:22.7386624Z Dynamo produced 4 graphs covering 1787 ops with 4 graph breaks (1 unique) 2025-12-04T09:39:26.4647798Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:39:26.4648751Z import pynvml # type: ignore[import] 2025-12-04T09:39:29.9147817Z 2025-12-04T09:39:32.2732783Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:39:32.2733193Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:39:32.2745481Z cpu eval BartForCausalLM 2025-12-04T09:39:33.9714519Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:34.6005053Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:35.2149256Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:42.1691745Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1692085Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1695487Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1696213Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1696520Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1696750Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1696983Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1697218Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1697439Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1697674Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1698223Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1698453Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1698732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1699172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1699563Z res = mod(**inputs) 2025-12-04T09:39:42.1700007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1700456Z outputs = self.model.decoder( 2025-12-04T09:39:42.1700893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1701322Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1701730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1702143Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1702596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1703021Z return func(*args, **kwargs) 2025-12-04T09:39:42.1703449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1703924Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1704362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1704793Z return func(*args, **kwargs) 2025-12-04T09:39:42.1705219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.1705867Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.1706097Z 2025-12-04T09:39:42.1706226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1706657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1707023Z res = mod(**inputs) 2025-12-04T09:39:42.1707433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1707871Z outputs = self.model.decoder( 2025-12-04T09:39:42.1708299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1708746Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1709149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1709574Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1710010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1710429Z return func(*args, **kwargs) 2025-12-04T09:39:42.1710842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1711309Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1711831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1712255Z return func(*args, **kwargs) 2025-12-04T09:39:42.1712663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.1713103Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.1713256Z 2025-12-04T09:39:42.1713384Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1713787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1714148Z res = mod(**inputs) 2025-12-04T09:39:42.1714604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1715033Z outputs = self.model.decoder( 2025-12-04T09:39:42.1715455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1715892Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1716291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1716697Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1717128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1717556Z return func(*args, **kwargs) 2025-12-04T09:39:42.1717979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1718444Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1718901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1719317Z return func(*args, **kwargs) 2025-12-04T09:39:42.1719729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.1720166Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.1720332Z 2025-12-04T09:39:42.1720423Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1721028Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1721432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1721799Z res = mod(**inputs) 2025-12-04T09:39:42.1722210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1722652Z outputs = self.model.decoder( 2025-12-04T09:39:42.1723075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1723510Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1723910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1724314Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1724742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1725162Z return func(*args, **kwargs) 2025-12-04T09:39:42.1725573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1726021Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1726476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1726888Z return func(*args, **kwargs) 2025-12-04T09:39:42.1727299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.1727825Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.1728332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.1728918Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.1729133Z 2025-12-04T09:39:42.1729253Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1729665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1730037Z res = mod(**inputs) 2025-12-04T09:39:42.1730444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1730974Z outputs = self.model.decoder( 2025-12-04T09:39:42.1731402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1731837Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1732234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1732654Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1733081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1733508Z return func(*args, **kwargs) 2025-12-04T09:39:42.1733920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1734377Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1734829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1735244Z return func(*args, **kwargs) 2025-12-04T09:39:42.1735653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.1736088Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.1736240Z 2025-12-04T09:39:42.1736361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1736753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1737117Z res = mod(**inputs) 2025-12-04T09:39:42.1737516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1737946Z outputs = self.model.decoder( 2025-12-04T09:39:42.1738369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1738793Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1739188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1739599Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1740015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1740434Z return func(*args, **kwargs) 2025-12-04T09:39:42.1740846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.1741322Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.1741525Z 2025-12-04T09:39:42.1741645Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1742058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1742418Z res = mod(**inputs) 2025-12-04T09:39:42.1742809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1743241Z outputs = self.model.decoder( 2025-12-04T09:39:42.1743728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1744153Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1744539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1744950Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1745362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1745825Z return func(*args, **kwargs) 2025-12-04T09:39:42.1746274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.1746752Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.1747429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.1747837Z return self.act(input) 2025-12-04T09:39:42.1747966Z 2025-12-04T09:39:42.1748085Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1748497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1748860Z res = mod(**inputs) 2025-12-04T09:39:42.1749253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1749690Z outputs = self.model.decoder( 2025-12-04T09:39:42.1750113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1750549Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1750940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1751348Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1751774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1752183Z return func(*args, **kwargs) 2025-12-04T09:39:42.1752597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.1753031Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.1753187Z 2025-12-04T09:39:42.1753312Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1753708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1754073Z res = mod(**inputs) 2025-12-04T09:39:42.1754472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1754892Z outputs = self.model.decoder( 2025-12-04T09:39:42.1755322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1755750Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1756145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1756546Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1756967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1757382Z return func(*args, **kwargs) 2025-12-04T09:39:42.1757797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1758253Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1758702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1759121Z return func(*args, **kwargs) 2025-12-04T09:39:42.1759683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.1760206Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.1760440Z 2025-12-04T09:39:42.1760652Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1761091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1761450Z res = mod(**inputs) 2025-12-04T09:39:42.1761868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1762370Z outputs = self.model.decoder( 2025-12-04T09:39:42.1762783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1763213Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1763611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1764024Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1764440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1764856Z return func(*args, **kwargs) 2025-12-04T09:39:42.1765268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1765736Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1766180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1766593Z return func(*args, **kwargs) 2025-12-04T09:39:42.1767000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.1767426Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.1767586Z 2025-12-04T09:39:42.1767702Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1768111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1768464Z res = mod(**inputs) 2025-12-04T09:39:42.1768855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1769286Z outputs = self.model.decoder( 2025-12-04T09:39:42.1769701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1770141Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1770525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1770934Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1771361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1771765Z return func(*args, **kwargs) 2025-12-04T09:39:42.1772178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1772646Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1773087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1773504Z return func(*args, **kwargs) 2025-12-04T09:39:42.1773916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.1774367Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.1774520Z 2025-12-04T09:39:42.1774609Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1774930Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1775334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1775693Z res = mod(**inputs) 2025-12-04T09:39:42.1776078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1776516Z outputs = self.model.decoder( 2025-12-04T09:39:42.1776940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1777359Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1778654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1779053Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1779480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1779891Z return func(*args, **kwargs) 2025-12-04T09:39:42.1780309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1780762Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1781209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1781615Z return func(*args, **kwargs) 2025-12-04T09:39:42.1782029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.1782494Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.1782985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.1783530Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.1783748Z 2025-12-04T09:39:42.1783868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1784269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1784623Z res = mod(**inputs) 2025-12-04T09:39:42.1785024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1785453Z outputs = self.model.decoder( 2025-12-04T09:39:42.1785871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1786293Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1786679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1787087Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1787504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1787918Z return func(*args, **kwargs) 2025-12-04T09:39:42.1788332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1788927Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1789372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1789797Z return func(*args, **kwargs) 2025-12-04T09:39:42.1790232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.1790681Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.1790845Z 2025-12-04T09:39:42.1790967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1791434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1791797Z res = mod(**inputs) 2025-12-04T09:39:42.1792191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1792638Z outputs = self.model.decoder( 2025-12-04T09:39:42.1793057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1793497Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1793882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1794339Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1794771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1795187Z return func(*args, **kwargs) 2025-12-04T09:39:42.1795615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.1796110Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.1796306Z 2025-12-04T09:39:42.1796434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1796834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1797206Z res = mod(**inputs) 2025-12-04T09:39:42.1797614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1798048Z outputs = self.model.decoder( 2025-12-04T09:39:42.1798478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1798918Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1799324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1799734Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1800171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1800670Z return func(*args, **kwargs) 2025-12-04T09:39:42.1801081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.1801561Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.1802000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.1802400Z return self.act(input) 2025-12-04T09:39:42.1802530Z 2025-12-04T09:39:42.1802649Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1803057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1803426Z res = mod(**inputs) 2025-12-04T09:39:42.1803833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1804259Z outputs = self.model.decoder( 2025-12-04T09:39:42.1804685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1805112Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1805501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1805924Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1806352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1806768Z return func(*args, **kwargs) 2025-12-04T09:39:42.1807238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.1807673Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.1807825Z 2025-12-04T09:39:42.1807949Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1808340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1808698Z res = mod(**inputs) 2025-12-04T09:39:42.1809087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1809511Z outputs = self.model.decoder( 2025-12-04T09:39:42.1809993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1810423Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1810847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1811263Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1811681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1812096Z return func(*args, **kwargs) 2025-12-04T09:39:42.1812506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1812951Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1813395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1813810Z return func(*args, **kwargs) 2025-12-04T09:39:42.1814220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.1814732Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.1814963Z 2025-12-04T09:39:42.1815088Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1815493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1815851Z res = mod(**inputs) 2025-12-04T09:39:42.1816243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1816668Z outputs = self.model.decoder( 2025-12-04T09:39:42.1817086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1817510Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1817899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1818316Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1818740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1819146Z return func(*args, **kwargs) 2025-12-04T09:39:42.1819561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1820019Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1820457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1820882Z return func(*args, **kwargs) 2025-12-04T09:39:42.1821298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.1821737Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.1821887Z 2025-12-04T09:39:42.1822006Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1822409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1822823Z res = mod(**inputs) 2025-12-04T09:39:42.1823221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1823655Z outputs = self.model.decoder( 2025-12-04T09:39:42.1824077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1824526Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1824908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1825370Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1825795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1826216Z return func(*args, **kwargs) 2025-12-04T09:39:42.1826622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1827075Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1827520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1827925Z return func(*args, **kwargs) 2025-12-04T09:39:42.1828338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.1828779Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.1828935Z 2025-12-04T09:39:42.1829033Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1829299Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1829701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1830066Z res = mod(**inputs) 2025-12-04T09:39:42.1830458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1830889Z outputs = self.model.decoder( 2025-12-04T09:39:42.1831306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1831734Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1832115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1832529Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1832958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1833375Z return func(*args, **kwargs) 2025-12-04T09:39:42.1833779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1834232Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1834680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1835081Z return func(*args, **kwargs) 2025-12-04T09:39:42.1835493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.1835949Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.1836449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.1836989Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.1837200Z 2025-12-04T09:39:42.1837316Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1837715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1838073Z res = mod(**inputs) 2025-12-04T09:39:42.1838516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1838960Z outputs = self.model.decoder( 2025-12-04T09:39:42.1839402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1839839Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1840252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1841343Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1841838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1842253Z return func(*args, **kwargs) 2025-12-04T09:39:42.1842672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1843143Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1843595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1844015Z return func(*args, **kwargs) 2025-12-04T09:39:42.1844436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.1844889Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.1845045Z 2025-12-04T09:39:42.1845166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1845579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1845944Z res = mod(**inputs) 2025-12-04T09:39:42.1846340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1846776Z outputs = self.model.decoder( 2025-12-04T09:39:42.1847432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1847871Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1848256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1848673Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1849097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1849519Z return func(*args, **kwargs) 2025-12-04T09:39:42.1849930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.1850410Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.1850602Z 2025-12-04T09:39:42.1850728Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1851122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1851485Z res = mod(**inputs) 2025-12-04T09:39:42.1851881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1852306Z outputs = self.model.decoder( 2025-12-04T09:39:42.1852716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1853142Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1853530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1853934Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1854346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1854885Z return func(*args, **kwargs) 2025-12-04T09:39:42.1855310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.1855785Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.1856228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.1856624Z return self.act(input) 2025-12-04T09:39:42.1856750Z 2025-12-04T09:39:42.1856876Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1857271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1857701Z res = mod(**inputs) 2025-12-04T09:39:42.1858100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1858521Z outputs = self.model.decoder( 2025-12-04T09:39:42.1858954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1859381Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1859775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1860172Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1860598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1861015Z return func(*args, **kwargs) 2025-12-04T09:39:42.1861423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.1861867Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.1862030Z 2025-12-04T09:39:42.1862147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1862552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1862906Z res = mod(**inputs) 2025-12-04T09:39:42.1863302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1863733Z outputs = self.model.decoder( 2025-12-04T09:39:42.1864156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1864579Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1864968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1865375Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1865788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1866205Z return func(*args, **kwargs) 2025-12-04T09:39:42.1866621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1867079Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1867520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1867934Z return func(*args, **kwargs) 2025-12-04T09:39:42.1868347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.1868856Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.1869097Z 2025-12-04T09:39:42.1869215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1869618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1869976Z res = mod(**inputs) 2025-12-04T09:39:42.1870417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1870852Z outputs = self.model.decoder( 2025-12-04T09:39:42.1871272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1871701Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1872085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1872490Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1872914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1873387Z return func(*args, **kwargs) 2025-12-04T09:39:42.1873798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1874252Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1874695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1875101Z return func(*args, **kwargs) 2025-12-04T09:39:42.1875508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.1875941Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.1876091Z 2025-12-04T09:39:42.1876214Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1876604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1876964Z res = mod(**inputs) 2025-12-04T09:39:42.1877355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1877773Z outputs = self.model.decoder( 2025-12-04T09:39:42.1878195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1878620Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1879009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1879410Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1879827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1880243Z return func(*args, **kwargs) 2025-12-04T09:39:42.1880874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1881348Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1881798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1882222Z return func(*args, **kwargs) 2025-12-04T09:39:42.1882627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.1883075Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.1883234Z 2025-12-04T09:39:42.1883338Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1883600Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1884009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1884372Z res = mod(**inputs) 2025-12-04T09:39:42.1884778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1885198Z outputs = self.model.decoder( 2025-12-04T09:39:42.1885620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1886103Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1886492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1886907Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1887334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1887755Z return func(*args, **kwargs) 2025-12-04T09:39:42.1888168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1888665Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1889107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1889521Z return func(*args, **kwargs) 2025-12-04T09:39:42.1889925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.1890381Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.1890877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.1891420Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.1891623Z 2025-12-04T09:39:42.1891739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1892141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1892505Z res = mod(**inputs) 2025-12-04T09:39:42.1892895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1893321Z outputs = self.model.decoder( 2025-12-04T09:39:42.1893747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1894172Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1894554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1894961Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1895382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1895790Z return func(*args, **kwargs) 2025-12-04T09:39:42.1896206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1896664Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1897109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1897512Z return func(*args, **kwargs) 2025-12-04T09:39:42.1897932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.1898372Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.1898523Z 2025-12-04T09:39:42.1898647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1899038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1899405Z res = mod(**inputs) 2025-12-04T09:39:42.1899800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1900247Z outputs = self.model.decoder( 2025-12-04T09:39:42.1900669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1901096Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1901529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1901929Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1902357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1902802Z return func(*args, **kwargs) 2025-12-04T09:39:42.1903213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.1903702Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.1903905Z 2025-12-04T09:39:42.1904066Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1904463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1904811Z res = mod(**inputs) 2025-12-04T09:39:42.1905207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1905640Z outputs = self.model.decoder( 2025-12-04T09:39:42.1906056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1906476Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1906863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1907262Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1907673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1908088Z return func(*args, **kwargs) 2025-12-04T09:39:42.1908495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.1908965Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.1909390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.1909783Z return self.act(input) 2025-12-04T09:39:42.1909908Z 2025-12-04T09:39:42.1910031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1910415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1910773Z res = mod(**inputs) 2025-12-04T09:39:42.1911164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1911588Z outputs = self.model.decoder( 2025-12-04T09:39:42.1912002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1912426Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1912813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1913215Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1913634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1914045Z return func(*args, **kwargs) 2025-12-04T09:39:42.1914453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.1914878Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.1915037Z 2025-12-04T09:39:42.1915150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1915552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1915907Z res = mod(**inputs) 2025-12-04T09:39:42.1916294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1916718Z outputs = self.model.decoder( 2025-12-04T09:39:42.1917174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1917599Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1917994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1918400Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1918823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1919233Z return func(*args, **kwargs) 2025-12-04T09:39:42.1919681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1920387Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1920927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1921350Z return func(*args, **kwargs) 2025-12-04T09:39:42.1921766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.1922288Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.1922519Z 2025-12-04T09:39:42.1922639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1923047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1923412Z res = mod(**inputs) 2025-12-04T09:39:42.1923822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1924246Z outputs = self.model.decoder( 2025-12-04T09:39:42.1924672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1925106Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1925497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1925915Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1926342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1926762Z return func(*args, **kwargs) 2025-12-04T09:39:42.1927167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1927626Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1928073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1928490Z return func(*args, **kwargs) 2025-12-04T09:39:42.1928896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.1929333Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.1929483Z 2025-12-04T09:39:42.1929608Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1929999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1930361Z res = mod(**inputs) 2025-12-04T09:39:42.1930762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1931194Z outputs = self.model.decoder( 2025-12-04T09:39:42.1931610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1932036Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1932426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1932905Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1933334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1933754Z return func(*args, **kwargs) 2025-12-04T09:39:42.1934169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1934617Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1935065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1935521Z return func(*args, **kwargs) 2025-12-04T09:39:42.1935935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.1936372Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.1936537Z 2025-12-04T09:39:42.1936633Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1936909Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1937304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1937670Z res = mod(**inputs) 2025-12-04T09:39:42.1938072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1938508Z outputs = self.model.decoder( 2025-12-04T09:39:42.1938924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1939360Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1939748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1940143Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1940575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1940992Z return func(*args, **kwargs) 2025-12-04T09:39:42.1941405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1941856Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1942303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1942719Z return func(*args, **kwargs) 2025-12-04T09:39:42.1943126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.1943583Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.1944079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.1944627Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.1944832Z 2025-12-04T09:39:42.1944950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1945356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1945708Z res = mod(**inputs) 2025-12-04T09:39:42.1946109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1946534Z outputs = self.model.decoder( 2025-12-04T09:39:42.1946966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1947529Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1947900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1948411Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1948831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1949253Z return func(*args, **kwargs) 2025-12-04T09:39:42.1949656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1950125Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1950568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1951044Z return func(*args, **kwargs) 2025-12-04T09:39:42.1951454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.1951877Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.1952025Z 2025-12-04T09:39:42.1952148Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1952551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1952903Z res = mod(**inputs) 2025-12-04T09:39:42.1953296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1953733Z outputs = self.model.decoder( 2025-12-04T09:39:42.1954142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1954579Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1954970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1955365Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1955772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1956175Z return func(*args, **kwargs) 2025-12-04T09:39:42.1956575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.1957029Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.1957223Z 2025-12-04T09:39:42.1957335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1957732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1958109Z res = mod(**inputs) 2025-12-04T09:39:42.1958520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1958947Z outputs = self.model.decoder( 2025-12-04T09:39:42.1959364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1959795Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1960264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1960828Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1961262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1961680Z return func(*args, **kwargs) 2025-12-04T09:39:42.1962101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.1962596Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.1963025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.1963415Z return self.act(input) 2025-12-04T09:39:42.1963546Z 2025-12-04T09:39:42.1963660Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1964111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1964458Z res = mod(**inputs) 2025-12-04T09:39:42.1964852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1965278Z outputs = self.model.decoder( 2025-12-04T09:39:42.1965693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1966118Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1966505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1966935Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1967333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1967739Z return func(*args, **kwargs) 2025-12-04T09:39:42.1968143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.1968561Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.1968706Z 2025-12-04T09:39:42.1968819Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1969203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1969545Z res = mod(**inputs) 2025-12-04T09:39:42.1969919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1970334Z outputs = self.model.decoder( 2025-12-04T09:39:42.1970738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1971150Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1971519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1971907Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1972310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1972705Z return func(*args, **kwargs) 2025-12-04T09:39:42.1973104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1973543Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1973974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1974369Z return func(*args, **kwargs) 2025-12-04T09:39:42.1974778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.1975287Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.1975512Z 2025-12-04T09:39:42.1975637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1976025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1976371Z res = mod(**inputs) 2025-12-04T09:39:42.1976758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1977172Z outputs = self.model.decoder( 2025-12-04T09:39:42.1977592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1978021Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1978409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1978804Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1979260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1979675Z return func(*args, **kwargs) 2025-12-04T09:39:42.1980097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1980552Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1980990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1981399Z return func(*args, **kwargs) 2025-12-04T09:39:42.1981875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.1982308Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.1982463Z 2025-12-04T09:39:42.1982798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1983200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1983548Z res = mod(**inputs) 2025-12-04T09:39:42.1983943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1984371Z outputs = self.model.decoder( 2025-12-04T09:39:42.1984787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1985215Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1985604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1986016Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1986427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1986840Z return func(*args, **kwargs) 2025-12-04T09:39:42.1987255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1987704Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1988136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1988543Z return func(*args, **kwargs) 2025-12-04T09:39:42.1988953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.1989395Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.1989557Z 2025-12-04T09:39:42.1989643Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1989915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1990309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1990655Z res = mod(**inputs) 2025-12-04T09:39:42.1991049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1991476Z outputs = self.model.decoder( 2025-12-04T09:39:42.1991879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.1992314Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.1992698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1993097Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1993515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1993919Z return func(*args, **kwargs) 2025-12-04T09:39:42.1994319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.1994822Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.1995263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1995685Z return func(*args, **kwargs) 2025-12-04T09:39:42.1996106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.1996572Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.1997082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.1997660Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.1997862Z 2025-12-04T09:39:42.1997986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1998375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1998737Z res = mod(**inputs) 2025-12-04T09:39:42.1999264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.1999703Z outputs = self.model.decoder( 2025-12-04T09:39:42.2000121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2000612Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2001049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2001459Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2001885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2002304Z return func(*args, **kwargs) 2025-12-04T09:39:42.2002707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2003149Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2003598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2004002Z return func(*args, **kwargs) 2025-12-04T09:39:42.2004391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.2004814Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.2004970Z 2025-12-04T09:39:42.2005086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2005470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2005808Z res = mod(**inputs) 2025-12-04T09:39:42.2006192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2006608Z outputs = self.model.decoder( 2025-12-04T09:39:42.2007008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2007417Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2007795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2008195Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2008609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2009023Z return func(*args, **kwargs) 2025-12-04T09:39:42.2009431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2009903Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2010087Z 2025-12-04T09:39:42.2010245Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2010647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2011001Z res = mod(**inputs) 2025-12-04T09:39:42.2011387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2011820Z outputs = self.model.decoder( 2025-12-04T09:39:42.2012244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2012711Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2013092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2013494Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2013920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2014335Z return func(*args, **kwargs) 2025-12-04T09:39:42.2014736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2015213Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2015641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.2016024Z return self.act(input) 2025-12-04T09:39:42.2016157Z 2025-12-04T09:39:42.2016273Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2016675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2017027Z res = mod(**inputs) 2025-12-04T09:39:42.2017413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2017836Z outputs = self.model.decoder( 2025-12-04T09:39:42.2018253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2018668Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2019059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2019460Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2019877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2020287Z return func(*args, **kwargs) 2025-12-04T09:39:42.2020698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.2021132Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.2021283Z 2025-12-04T09:39:42.2021405Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2021795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2022152Z res = mod(**inputs) 2025-12-04T09:39:42.2022542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2022961Z outputs = self.model.decoder( 2025-12-04T09:39:42.2023380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2023806Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2024194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2024594Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2025008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2025465Z return func(*args, **kwargs) 2025-12-04T09:39:42.2025858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2026296Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2026733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2027140Z return func(*args, **kwargs) 2025-12-04T09:39:42.2027540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.2028087Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.2028312Z 2025-12-04T09:39:42.2028436Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2028834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2029191Z res = mod(**inputs) 2025-12-04T09:39:42.2029590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2030022Z outputs = self.model.decoder( 2025-12-04T09:39:42.2030439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2030859Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2031242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2031667Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2032092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2032518Z return func(*args, **kwargs) 2025-12-04T09:39:42.2032933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2033379Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2033833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2034251Z return func(*args, **kwargs) 2025-12-04T09:39:42.2034676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.2035095Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.2035252Z 2025-12-04T09:39:42.2035365Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2035758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2036134Z res = mod(**inputs) 2025-12-04T09:39:42.2036546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2036984Z outputs = self.model.decoder( 2025-12-04T09:39:42.2037405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2037834Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2038222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2038635Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2039057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2039469Z return func(*args, **kwargs) 2025-12-04T09:39:42.2039878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2040345Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2040913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2041334Z return func(*args, **kwargs) 2025-12-04T09:39:42.2041742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.2042193Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.2042352Z 2025-12-04T09:39:42.2042442Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2042715Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2043116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2043516Z res = mod(**inputs) 2025-12-04T09:39:42.2043912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2044343Z outputs = self.model.decoder( 2025-12-04T09:39:42.2044768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2045188Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2045581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2045985Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2046407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2046816Z return func(*args, **kwargs) 2025-12-04T09:39:42.2047414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2047881Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2048324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2048742Z return func(*args, **kwargs) 2025-12-04T09:39:42.2049152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.2049604Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2050097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2050640Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2050844Z 2025-12-04T09:39:42.2050971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2051379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2051730Z res = mod(**inputs) 2025-12-04T09:39:42.2052131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2052561Z outputs = self.model.decoder( 2025-12-04T09:39:42.2052977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2053395Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2053779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2054174Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2054577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2054980Z return func(*args, **kwargs) 2025-12-04T09:39:42.2055388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2055845Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2056287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2056801Z return func(*args, **kwargs) 2025-12-04T09:39:42.2057206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.2057624Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.2057782Z 2025-12-04T09:39:42.2057896Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2058289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2058657Z res = mod(**inputs) 2025-12-04T09:39:42.2059053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2059533Z outputs = self.model.decoder( 2025-12-04T09:39:42.2060013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2060430Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2060816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2061209Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2061621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2062073Z return func(*args, **kwargs) 2025-12-04T09:39:42.2062501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2062995Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2063190Z 2025-12-04T09:39:42.2063307Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2063708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2064065Z res = mod(**inputs) 2025-12-04T09:39:42.2064464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2064894Z outputs = self.model.decoder( 2025-12-04T09:39:42.2065323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2065760Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2066147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2066548Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2066983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2067398Z return func(*args, **kwargs) 2025-12-04T09:39:42.2067811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2068296Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2068729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.2069109Z return self.act(input) 2025-12-04T09:39:42.2069234Z 2025-12-04T09:39:42.2069349Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2069761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2070129Z res = mod(**inputs) 2025-12-04T09:39:42.2070525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2070965Z outputs = self.model.decoder( 2025-12-04T09:39:42.2071389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2071826Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2072254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2072673Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2073118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2073535Z return func(*args, **kwargs) 2025-12-04T09:39:42.2073951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.2074391Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.2074589Z 2025-12-04T09:39:42.2074714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2075105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2075461Z res = mod(**inputs) 2025-12-04T09:39:42.2075859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2076286Z outputs = self.model.decoder( 2025-12-04T09:39:42.2076697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2077120Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2077508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2077913Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2078334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2078753Z return func(*args, **kwargs) 2025-12-04T09:39:42.2079165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2079624Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2080073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2080556Z return func(*args, **kwargs) 2025-12-04T09:39:42.2080974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.2081497Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.2081731Z 2025-12-04T09:39:42.2081848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2082246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2082599Z res = mod(**inputs) 2025-12-04T09:39:42.2082996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2083434Z outputs = self.model.decoder( 2025-12-04T09:39:42.2083853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2084272Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2084665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2085079Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2085491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2085906Z return func(*args, **kwargs) 2025-12-04T09:39:42.2086311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2086760Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2087191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2087649Z return func(*args, **kwargs) 2025-12-04T09:39:42.2088064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.2088499Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.2088650Z 2025-12-04T09:39:42.2088766Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2089167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2089525Z res = mod(**inputs) 2025-12-04T09:39:42.2089920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2090389Z outputs = self.model.decoder( 2025-12-04T09:39:42.2090807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2091233Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2091616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2092021Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2092441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2092847Z return func(*args, **kwargs) 2025-12-04T09:39:42.2093258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2093711Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2094156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2094564Z return func(*args, **kwargs) 2025-12-04T09:39:42.2094972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.2095416Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.2095573Z 2025-12-04T09:39:42.2095673Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2095934Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2096330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2096704Z res = mod(**inputs) 2025-12-04T09:39:42.2097091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2097533Z outputs = self.model.decoder( 2025-12-04T09:39:42.2097957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2098394Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2098774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2099181Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2099603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2100016Z return func(*args, **kwargs) 2025-12-04T09:39:42.2100428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2100889Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2101328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2101768Z return func(*args, **kwargs) 2025-12-04T09:39:42.2102179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.2102645Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2103172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2103722Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2103933Z 2025-12-04T09:39:42.2104050Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2104445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2104813Z res = mod(**inputs) 2025-12-04T09:39:42.2105206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2105685Z outputs = self.model.decoder( 2025-12-04T09:39:42.2106101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2106522Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2106912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2107311Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2107720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2108135Z return func(*args, **kwargs) 2025-12-04T09:39:42.2108545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2108991Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2109422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2109840Z return func(*args, **kwargs) 2025-12-04T09:39:42.2110247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.2110682Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.2110836Z 2025-12-04T09:39:42.2110953Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2111351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2111708Z res = mod(**inputs) 2025-12-04T09:39:42.2112095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2112523Z outputs = self.model.decoder( 2025-12-04T09:39:42.2112937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2113367Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2113746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2114149Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2114568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2114971Z return func(*args, **kwargs) 2025-12-04T09:39:42.2115380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2115852Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2116041Z 2025-12-04T09:39:42.2116163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2116550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2116919Z res = mod(**inputs) 2025-12-04T09:39:42.2117315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2117752Z outputs = self.model.decoder( 2025-12-04T09:39:42.2118197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2118626Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2119016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2119418Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2119698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2119777Z return func(*args, **kwargs) 2025-12-04T09:39:42.2120056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2120231Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2120554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.2120653Z return self.act(input) 2025-12-04T09:39:42.2120657Z 2025-12-04T09:39:42.2120780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2121002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2121084Z res = mod(**inputs) 2025-12-04T09:39:42.2121363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2121454Z outputs = self.model.decoder( 2025-12-04T09:39:42.2121732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2121814Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2122072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2122162Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2122439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2122519Z return func(*args, **kwargs) 2025-12-04T09:39:42.2122795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.2122893Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.2122897Z 2025-12-04T09:39:42.2123013Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2123232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2123316Z res = mod(**inputs) 2025-12-04T09:39:42.2123593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2123685Z outputs = self.model.decoder( 2025-12-04T09:39:42.2123957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2124038Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2124303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2124391Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2124653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2124739Z return func(*args, **kwargs) 2025-12-04T09:39:42.2125011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2125130Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2125398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2125475Z return func(*args, **kwargs) 2025-12-04T09:39:42.2125799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.2125971Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.2125975Z 2025-12-04T09:39:42.2126095Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2126313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2126386Z res = mod(**inputs) 2025-12-04T09:39:42.2126673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2126753Z outputs = self.model.decoder( 2025-12-04T09:39:42.2127090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2127178Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2127431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2127529Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2127816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2127893Z return func(*args, **kwargs) 2025-12-04T09:39:42.2128183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2128294Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2128587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2128665Z return func(*args, **kwargs) 2025-12-04T09:39:42.2128952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.2129052Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.2129056Z 2025-12-04T09:39:42.2129173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2129396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2129476Z res = mod(**inputs) 2025-12-04T09:39:42.2129768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2129858Z outputs = self.model.decoder( 2025-12-04T09:39:42.2130149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2130229Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2130491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2130579Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2130859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2130947Z return func(*args, **kwargs) 2025-12-04T09:39:42.2131237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2131354Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2131636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2131723Z return func(*args, **kwargs) 2025-12-04T09:39:42.2132020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.2132120Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.2132124Z 2025-12-04T09:39:42.2132223Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2132336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2132615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2132697Z res = mod(**inputs) 2025-12-04T09:39:42.2132991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2133072Z outputs = self.model.decoder( 2025-12-04T09:39:42.2133365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2133444Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2133698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2133823Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2134100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2134184Z return func(*args, **kwargs) 2025-12-04T09:39:42.2134462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2134572Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2134863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2134938Z return func(*args, **kwargs) 2025-12-04T09:39:42.2135227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.2135338Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2135670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2135832Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2135835Z 2025-12-04T09:39:42.2135951Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2136189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2136262Z res = mod(**inputs) 2025-12-04T09:39:42.2136542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2136632Z outputs = self.model.decoder( 2025-12-04T09:39:42.2136934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2137014Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2137275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2137364Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2137641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2137719Z return func(*args, **kwargs) 2025-12-04T09:39:42.2137997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2138115Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2138382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2138466Z return func(*args, **kwargs) 2025-12-04T09:39:42.2138742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.2138837Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.2138841Z 2025-12-04T09:39:42.2138962Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2139181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2139252Z res = mod(**inputs) 2025-12-04T09:39:42.2139591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2139674Z outputs = self.model.decoder( 2025-12-04T09:39:42.2139956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2140036Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2140282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2140375Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2140676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2140759Z return func(*args, **kwargs) 2025-12-04T09:39:42.2141033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2141167Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2141171Z 2025-12-04T09:39:42.2141294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2141511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2141582Z res = mod(**inputs) 2025-12-04T09:39:42.2141864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2141943Z outputs = self.model.decoder( 2025-12-04T09:39:42.2142225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2142306Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2142552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2142646Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2142914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2142989Z return func(*args, **kwargs) 2025-12-04T09:39:42.2143269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2143400Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2143645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.2143722Z return self.act(input) 2025-12-04T09:39:42.2143729Z 2025-12-04T09:39:42.2143841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2144069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2144141Z res = mod(**inputs) 2025-12-04T09:39:42.2144429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2144509Z outputs = self.model.decoder( 2025-12-04T09:39:42.2144783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2144872Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2145115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2145205Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2145478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2145558Z return func(*args, **kwargs) 2025-12-04T09:39:42.2145837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.2145927Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.2145969Z 2025-12-04T09:39:42.2146086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2146313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2146383Z res = mod(**inputs) 2025-12-04T09:39:42.2146670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2146750Z outputs = self.model.decoder( 2025-12-04T09:39:42.2147030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2147468Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2147720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2147808Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2148093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2148172Z return func(*args, **kwargs) 2025-12-04T09:39:42.2148459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2148570Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2148839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2148928Z return func(*args, **kwargs) 2025-12-04T09:39:42.2149202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.2149373Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.2149385Z 2025-12-04T09:39:42.2149499Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2149718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2149798Z res = mod(**inputs) 2025-12-04T09:39:42.2150076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2150157Z outputs = self.model.decoder( 2025-12-04T09:39:42.2150444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2150523Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2150779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2150868Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2151137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2151222Z return func(*args, **kwargs) 2025-12-04T09:39:42.2151503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2151612Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2151889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2151966Z return func(*args, **kwargs) 2025-12-04T09:39:42.2152251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.2152339Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.2152346Z 2025-12-04T09:39:42.2152458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2152684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2152756Z res = mod(**inputs) 2025-12-04T09:39:42.2153123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2153207Z outputs = self.model.decoder( 2025-12-04T09:39:42.2153488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2153574Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2153822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2153909Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2154186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2154306Z return func(*args, **kwargs) 2025-12-04T09:39:42.2154590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2154699Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2154967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2155051Z return func(*args, **kwargs) 2025-12-04T09:39:42.2155323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.2155420Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.2155431Z 2025-12-04T09:39:42.2155522Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2155634Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2155862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2155934Z res = mod(**inputs) 2025-12-04T09:39:42.2156211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2156300Z outputs = self.model.decoder( 2025-12-04T09:39:42.2156575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2156665Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2156909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2156995Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2157268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2157343Z return func(*args, **kwargs) 2025-12-04T09:39:42.2157618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2157737Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2158003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2158086Z return func(*args, **kwargs) 2025-12-04T09:39:42.2158355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.2158465Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2158798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2158949Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2158953Z 2025-12-04T09:39:42.2159077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2159296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2159369Z res = mod(**inputs) 2025-12-04T09:39:42.2159700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2159783Z outputs = self.model.decoder( 2025-12-04T09:39:42.2160264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2160360Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2160673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2160775Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2161047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2161172Z return func(*args, **kwargs) 2025-12-04T09:39:42.2161454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2161563Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2161831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2161913Z return func(*args, **kwargs) 2025-12-04T09:39:42.2162190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.2162290Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.2162294Z 2025-12-04T09:39:42.2162410Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2162628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2162712Z res = mod(**inputs) 2025-12-04T09:39:42.2162987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2163078Z outputs = self.model.decoder( 2025-12-04T09:39:42.2163358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2163439Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2163695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2163782Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2164049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2164131Z return func(*args, **kwargs) 2025-12-04T09:39:42.2164403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2164546Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2164550Z 2025-12-04T09:39:42.2164687Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2164902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2164987Z res = mod(**inputs) 2025-12-04T09:39:42.2165261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2165350Z outputs = self.model.decoder( 2025-12-04T09:39:42.2165624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2165704Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2165957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2166048Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2166313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2166399Z return func(*args, **kwargs) 2025-12-04T09:39:42.2166772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2166910Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2167157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.2167235Z return self.act(input) 2025-12-04T09:39:42.2167238Z 2025-12-04T09:39:42.2167358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2167588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2167660Z res = mod(**inputs) 2025-12-04T09:39:42.2167990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2168072Z outputs = self.model.decoder( 2025-12-04T09:39:42.2168363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2168446Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2168692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2168787Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2169066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2169149Z return func(*args, **kwargs) 2025-12-04T09:39:42.2169427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.2169520Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.2169523Z 2025-12-04T09:39:42.2169643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2169872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2169942Z res = mod(**inputs) 2025-12-04T09:39:42.2170237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2170319Z outputs = self.model.decoder( 2025-12-04T09:39:42.2170616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2170694Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2170941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2171033Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2171323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2171399Z return func(*args, **kwargs) 2025-12-04T09:39:42.2171694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2171803Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2172092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2172167Z return func(*args, **kwargs) 2025-12-04T09:39:42.2172453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.2172628Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.2172631Z 2025-12-04T09:39:42.2172745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2172971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2173042Z res = mod(**inputs) 2025-12-04T09:39:42.2173330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2173459Z outputs = self.model.decoder( 2025-12-04T09:39:42.2173746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2173826Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2174078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2174163Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2174444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2174557Z return func(*args, **kwargs) 2025-12-04T09:39:42.2174830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2174945Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2175224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2175305Z return func(*args, **kwargs) 2025-12-04T09:39:42.2175590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.2175678Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.2175682Z 2025-12-04T09:39:42.2175804Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2176028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2176099Z res = mod(**inputs) 2025-12-04T09:39:42.2176398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2176481Z outputs = self.model.decoder( 2025-12-04T09:39:42.2176762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2176844Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2177089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2177184Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2177452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2177536Z return func(*args, **kwargs) 2025-12-04T09:39:42.2177810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2177966Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2178294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2178374Z return func(*args, **kwargs) 2025-12-04T09:39:42.2178648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.2178752Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.2178756Z 2025-12-04T09:39:42.2178847Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2178968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2179182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2179253Z res = mod(**inputs) 2025-12-04T09:39:42.2179530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2179615Z outputs = self.model.decoder( 2025-12-04T09:39:42.2179889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2179976Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2180266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2180363Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2180633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2180710Z return func(*args, **kwargs) 2025-12-04T09:39:42.2180989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2181096Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2181367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2181478Z return func(*args, **kwargs) 2025-12-04T09:39:42.2181750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.2181868Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2182193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2182346Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2182358Z 2025-12-04T09:39:42.2182471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2182690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2182769Z res = mod(**inputs) 2025-12-04T09:39:42.2183045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2183129Z outputs = self.model.decoder( 2025-12-04T09:39:42.2183411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2183491Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2183744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2183831Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2184096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2184180Z return func(*args, **kwargs) 2025-12-04T09:39:42.2184451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2184558Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2184838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2184915Z return func(*args, **kwargs) 2025-12-04T09:39:42.2185195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.2185289Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.2185293Z 2025-12-04T09:39:42.2185406Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2185634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2185706Z res = mod(**inputs) 2025-12-04T09:39:42.2185979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2186069Z outputs = self.model.decoder( 2025-12-04T09:39:42.2186345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2186436Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2186683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2186770Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2187084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2187164Z return func(*args, **kwargs) 2025-12-04T09:39:42.2187447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2187576Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2187580Z 2025-12-04T09:39:42.2187697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2187924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2188046Z res = mod(**inputs) 2025-12-04T09:39:42.2188324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2188411Z outputs = self.model.decoder( 2025-12-04T09:39:42.2188689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2188776Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2189025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2189113Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2189387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2189462Z return func(*args, **kwargs) 2025-12-04T09:39:42.2189741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2189874Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2190111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.2190197Z return self.act(input) 2025-12-04T09:39:42.2190201Z 2025-12-04T09:39:42.2190314Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2190533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2190611Z res = mod(**inputs) 2025-12-04T09:39:42.2190893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2190979Z outputs = self.model.decoder( 2025-12-04T09:39:42.2191256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2191338Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2191591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2191678Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2191948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2192032Z return func(*args, **kwargs) 2025-12-04T09:39:42.2192305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.2192404Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.2192408Z 2025-12-04T09:39:42.2192521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2192738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2192823Z res = mod(**inputs) 2025-12-04T09:39:42.2193098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2193186Z outputs = self.model.decoder( 2025-12-04T09:39:42.2193551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2193632Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2193884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2193972Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2194240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2194327Z return func(*args, **kwargs) 2025-12-04T09:39:42.2194598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2194750Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2195023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2195099Z return func(*args, **kwargs) 2025-12-04T09:39:42.2195384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:39:42.2195553Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:39:42.2195558Z 2025-12-04T09:39:42.2195683Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2195900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2195972Z res = mod(**inputs) 2025-12-04T09:39:42.2196258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2196342Z outputs = self.model.decoder( 2025-12-04T09:39:42.2196622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2196715Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2196965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2197062Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2197329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2197405Z return func(*args, **kwargs) 2025-12-04T09:39:42.2197684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2197794Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2198080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2198165Z return func(*args, **kwargs) 2025-12-04T09:39:42.2198440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:39:42.2198538Z key_states = self.k_proj(current_states) 2025-12-04T09:39:42.2198542Z 2025-12-04T09:39:42.2198657Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2198874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2198953Z res = mod(**inputs) 2025-12-04T09:39:42.2199252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2199341Z outputs = self.model.decoder( 2025-12-04T09:39:42.2199616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2199698Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2199952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2200039Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2200341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2200428Z return func(*args, **kwargs) 2025-12-04T09:39:42.2200818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2200940Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2201210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2201286Z return func(*args, **kwargs) 2025-12-04T09:39:42.2201610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:39:42.2201708Z value_states = self.v_proj(current_states) 2025-12-04T09:39:42.2201712Z 2025-12-04T09:39:42.2201811Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2201926Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2202148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2202231Z res = mod(**inputs) 2025-12-04T09:39:42.2202507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2202590Z outputs = self.model.decoder( 2025-12-04T09:39:42.2202871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2202952Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2203207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2203293Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2203558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2203643Z return func(*args, **kwargs) 2025-12-04T09:39:42.2203916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2204026Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2204297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2204373Z return func(*args, **kwargs) 2025-12-04T09:39:42.2204677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:39:42.2204790Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2205110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2205264Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2205268Z 2025-12-04T09:39:42.2205383Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2205607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2205678Z res = mod(**inputs) 2025-12-04T09:39:42.2205953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2206040Z outputs = self.model.decoder( 2025-12-04T09:39:42.2206311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2206395Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2206647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2206734Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2207048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2207129Z return func(*args, **kwargs) 2025-12-04T09:39:42.2207400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:39:42.2207516Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:39:42.2207790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2207876Z return func(*args, **kwargs) 2025-12-04T09:39:42.2208160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:39:42.2208285Z attn_output = self.out_proj(attn_output) 2025-12-04T09:39:42.2208289Z 2025-12-04T09:39:42.2208408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2208635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2208711Z res = mod(**inputs) 2025-12-04T09:39:42.2208996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2209077Z outputs = self.model.decoder( 2025-12-04T09:39:42.2209359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2209440Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2209689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2209790Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2210065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2210145Z return func(*args, **kwargs) 2025-12-04T09:39:42.2210433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2210565Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2210570Z 2025-12-04T09:39:42.2210691Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2210922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2210997Z res = mod(**inputs) 2025-12-04T09:39:42.2211284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2211364Z outputs = self.model.decoder( 2025-12-04T09:39:42.2211665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2211746Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2211992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2212091Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2212376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2212451Z return func(*args, **kwargs) 2025-12-04T09:39:42.2212742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:39:42.2212871Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:39:42.2213124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:42.2213206Z return self.act(input) 2025-12-04T09:39:42.2213210Z 2025-12-04T09:39:42.2213322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2213548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2213620Z res = mod(**inputs) 2025-12-04T09:39:42.2213957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:39:42.2214041Z outputs = self.model.decoder( 2025-12-04T09:39:42.2214327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:39:42.2214413Z layer_outputs = decoder_layer( 2025-12-04T09:39:42.2214656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2214744Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2215077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2215154Z return func(*args, **kwargs) 2025-12-04T09:39:42.2215432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:39:42.2215525Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:39:42.2215530Z 2025-12-04T09:39:42.2215642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2215868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2215941Z res = mod(**inputs) 2025-12-04T09:39:42.2216215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1918, in forward 2025-12-04T09:39:42.2216309Z logits = self.lm_head(outputs[0]) 2025-12-04T09:39:42.2216313Z 2025-12-04T09:39:42.2216428Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2216654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2216725Z res = mod(**inputs) 2025-12-04T09:39:42.2217012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1924, in forward 2025-12-04T09:39:42.2217184Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:39:42.2217188Z 2025-12-04T09:39:53.4087203Z Compilation time (from dynamo_timed): 17.00686492 2025-12-04T09:39:53.4380966Z pass 2025-12-04T09:39:53.4381693Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:53.4382759Z TIMING: _recursive_pre_grad_passes:0.00642 _recursive_joint_graph_passes:0.65181 _recursive_post_grad_passes:0.06513 async_compile.wait:0.90629 code_gen:9.92754 inductor_compile:11.20183 backend_compile:14.2915 gc:0.00018 entire_frame_compile:17.00686 total_wall_time:17.00686 2025-12-04T09:39:53.4384071Z STATS: call_* op count: 336 | FakeTensorMode.__torch_dispatch__:7356 | FakeTensor.__torch_dispatch__:4394 | ProxyTorchDispatchMode.__torch_dispatch__:2034 2025-12-04T09:39:53.4384655Z Dynamo produced 1 graphs covering 336 ops with 0 graph breaks (0 unique) 2025-12-04T09:39:55.8398109Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:39:55.8399137Z import pynvml # type: ignore[import] 2025-12-04T09:39:59.3019614Z 2025-12-04T09:40:00.3738929Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:40:00.3739246Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:40:00.3757634Z cpu eval BertForMaskedLM 2025-12-04T09:40:00.8620431Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:01.1144472Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:01.3643582Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:08.9629165Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9629570Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9629813Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9630048Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9630277Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9630510Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9630739Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9630968Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9631194Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9631421Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9631781Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9632027Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9632298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9632776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9633155Z res = mod(**inputs) 2025-12-04T09:40:08.9633622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9634077Z outputs = self.bert( 2025-12-04T09:40:08.9634514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9634956Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9635400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9635846Z layer_outputs = layer_module( 2025-12-04T09:40:08.9636248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9636701Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9637151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9637587Z return func(*args, **kwargs) 2025-12-04T09:40:08.9638015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9638469Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9638913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9639340Z return func(*args, **kwargs) 2025-12-04T09:40:08.9639762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9640199Z self_outputs = self.self( 2025-12-04T09:40:08.9640810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9641242Z return func(*args, **kwargs) 2025-12-04T09:40:08.9641670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:08.9642295Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:08.9642598Z 2025-12-04T09:40:08.9642731Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9643139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9643513Z res = mod(**inputs) 2025-12-04T09:40:08.9643917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9644355Z outputs = self.bert( 2025-12-04T09:40:08.9644766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9645206Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9645683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9646107Z layer_outputs = layer_module( 2025-12-04T09:40:08.9646497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9646922Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9647554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9648039Z return func(*args, **kwargs) 2025-12-04T09:40:08.9648519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9649086Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9649500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9649908Z return func(*args, **kwargs) 2025-12-04T09:40:08.9650316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9650731Z self_outputs = self.self( 2025-12-04T09:40:08.9651123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9651535Z return func(*args, **kwargs) 2025-12-04T09:40:08.9651944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:08.9652357Z self.key(current_states) 2025-12-04T09:40:08.9652488Z 2025-12-04T09:40:08.9652603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9652998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9653346Z res = mod(**inputs) 2025-12-04T09:40:08.9653734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9654205Z outputs = self.bert( 2025-12-04T09:40:08.9654616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9655053Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9655462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9655872Z layer_outputs = layer_module( 2025-12-04T09:40:08.9656247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9656637Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9657050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9657454Z return func(*args, **kwargs) 2025-12-04T09:40:08.9657850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9658276Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9658683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9659089Z return func(*args, **kwargs) 2025-12-04T09:40:08.9659476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9659889Z self_outputs = self.self( 2025-12-04T09:40:08.9660275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9660673Z return func(*args, **kwargs) 2025-12-04T09:40:08.9661059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:08.9661479Z self.value(current_states) 2025-12-04T09:40:08.9661667Z 2025-12-04T09:40:08.9661766Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9662028Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9662413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9662759Z res = mod(**inputs) 2025-12-04T09:40:08.9663148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9663561Z outputs = self.bert( 2025-12-04T09:40:08.9663949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9664436Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9664852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9665255Z layer_outputs = layer_module( 2025-12-04T09:40:08.9665634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9666026Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9666433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9666847Z return func(*args, **kwargs) 2025-12-04T09:40:08.9667243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9667676Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9668086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9668490Z return func(*args, **kwargs) 2025-12-04T09:40:08.9668889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9669295Z self_outputs = self.self( 2025-12-04T09:40:08.9669678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9670078Z return func(*args, **kwargs) 2025-12-04T09:40:08.9670477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:08.9670948Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:08.9671158Z 2025-12-04T09:40:08.9671275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9671674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9672022Z res = mod(**inputs) 2025-12-04T09:40:08.9672407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9672811Z outputs = self.bert( 2025-12-04T09:40:08.9673201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9673610Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9674020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9674427Z layer_outputs = layer_module( 2025-12-04T09:40:08.9674802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9675185Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9675592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9675988Z return func(*args, **kwargs) 2025-12-04T09:40:08.9676388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9676843Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9677270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9677681Z return func(*args, **kwargs) 2025-12-04T09:40:08.9678088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:08.9678576Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:08.9679400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:08.9679918Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9680073Z 2025-12-04T09:40:08.9680192Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9680702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9681079Z res = mod(**inputs) 2025-12-04T09:40:08.9681481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9681907Z outputs = self.bert( 2025-12-04T09:40:08.9682305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9682731Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9683139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9683565Z layer_outputs = layer_module( 2025-12-04T09:40:08.9683954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9684360Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9684780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9685202Z return func(*args, **kwargs) 2025-12-04T09:40:08.9685613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9686061Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9686509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9686958Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9687413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:08.9687919Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:08.9688394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:08.9688909Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9689060Z 2025-12-04T09:40:08.9689183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9689587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9689957Z res = mod(**inputs) 2025-12-04T09:40:08.9690353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9690773Z outputs = self.bert( 2025-12-04T09:40:08.9691183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9691615Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9692038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9692472Z layer_outputs = layer_module( 2025-12-04T09:40:08.9692893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9693293Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9693709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9694110Z return func(*args, **kwargs) 2025-12-04T09:40:08.9694516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9694949Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9695388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9695909Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9696348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:08.9696856Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:08.9697326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:08.9697800Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:08.9698220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:08.9698608Z return self.act(input) 2025-12-04T09:40:08.9698730Z 2025-12-04T09:40:08.9698843Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9699255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9699629Z res = mod(**inputs) 2025-12-04T09:40:08.9700037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9700452Z outputs = self.bert( 2025-12-04T09:40:08.9700848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9701275Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9701686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9702106Z layer_outputs = layer_module( 2025-12-04T09:40:08.9702489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9702889Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9703309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9703720Z return func(*args, **kwargs) 2025-12-04T09:40:08.9704130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9704571Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9705019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9705458Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9705906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:08.9706423Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:08.9706903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:08.9707339Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9707489Z 2025-12-04T09:40:08.9707605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9708000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9708401Z res = mod(**inputs) 2025-12-04T09:40:08.9708793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9709201Z outputs = self.bert( 2025-12-04T09:40:08.9709584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9710013Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9710412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9710859Z layer_outputs = layer_module( 2025-12-04T09:40:08.9711238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9711631Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9712036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9712431Z return func(*args, **kwargs) 2025-12-04T09:40:08.9712828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9713251Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9713653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9714049Z return func(*args, **kwargs) 2025-12-04T09:40:08.9714443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9714949Z self_outputs = self.self( 2025-12-04T09:40:08.9715386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9715789Z return func(*args, **kwargs) 2025-12-04T09:40:08.9716187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:08.9716741Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:08.9717035Z 2025-12-04T09:40:08.9717149Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9717542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9717883Z res = mod(**inputs) 2025-12-04T09:40:08.9718272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9718678Z outputs = self.bert( 2025-12-04T09:40:08.9719073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9719489Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9719909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9720328Z layer_outputs = layer_module( 2025-12-04T09:40:08.9720768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9721179Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9721601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9722022Z return func(*args, **kwargs) 2025-12-04T09:40:08.9722423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9722861Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9723292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9723769Z return func(*args, **kwargs) 2025-12-04T09:40:08.9724174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9724594Z self_outputs = self.self( 2025-12-04T09:40:08.9724997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9725409Z return func(*args, **kwargs) 2025-12-04T09:40:08.9725819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:08.9726246Z self.key(current_states) 2025-12-04T09:40:08.9726413Z 2025-12-04T09:40:08.9726537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9726933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9727293Z res = mod(**inputs) 2025-12-04T09:40:08.9727697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9728109Z outputs = self.bert( 2025-12-04T09:40:08.9728531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9728965Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9729385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9729810Z layer_outputs = layer_module( 2025-12-04T09:40:08.9730198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9730607Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9731026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9731426Z return func(*args, **kwargs) 2025-12-04T09:40:08.9731836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9732275Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9732693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9733104Z return func(*args, **kwargs) 2025-12-04T09:40:08.9733509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9733927Z self_outputs = self.self( 2025-12-04T09:40:08.9734325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9734735Z return func(*args, **kwargs) 2025-12-04T09:40:08.9735144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:08.9735559Z self.value(current_states) 2025-12-04T09:40:08.9735697Z 2025-12-04T09:40:08.9735786Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9736060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9736445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9736783Z res = mod(**inputs) 2025-12-04T09:40:08.9737170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9737575Z outputs = self.bert( 2025-12-04T09:40:08.9737957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9738368Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9738773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9739219Z layer_outputs = layer_module( 2025-12-04T09:40:08.9739595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9739986Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9740399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9740803Z return func(*args, **kwargs) 2025-12-04T09:40:08.9741193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9741646Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9742061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9742461Z return func(*args, **kwargs) 2025-12-04T09:40:08.9742857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9743262Z self_outputs = self.self( 2025-12-04T09:40:08.9743669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9744075Z return func(*args, **kwargs) 2025-12-04T09:40:08.9744459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:08.9744945Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:08.9745152Z 2025-12-04T09:40:08.9745266Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9745658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9745998Z res = mod(**inputs) 2025-12-04T09:40:08.9746382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9746793Z outputs = self.bert( 2025-12-04T09:40:08.9747359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9747802Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9748211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9748632Z layer_outputs = layer_module( 2025-12-04T09:40:08.9749003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9749397Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9749809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9750228Z return func(*args, **kwargs) 2025-12-04T09:40:08.9750630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9751067Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9751488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9751904Z return func(*args, **kwargs) 2025-12-04T09:40:08.9752299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:08.9752765Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:08.9753226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:08.9753643Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9753799Z 2025-12-04T09:40:08.9753916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9754378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9754723Z res = mod(**inputs) 2025-12-04T09:40:08.9755107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9755514Z outputs = self.bert( 2025-12-04T09:40:08.9755902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9756305Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9756714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9757186Z layer_outputs = layer_module( 2025-12-04T09:40:08.9757561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9757939Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9758349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9758750Z return func(*args, **kwargs) 2025-12-04T09:40:08.9759138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9759566Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9760011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9760451Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9760955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:08.9761469Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:08.9761945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:08.9762387Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9762538Z 2025-12-04T09:40:08.9762653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9763054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9763409Z res = mod(**inputs) 2025-12-04T09:40:08.9763796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9764230Z outputs = self.bert( 2025-12-04T09:40:08.9764633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9765062Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9765472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9765898Z layer_outputs = layer_module( 2025-12-04T09:40:08.9766291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9766687Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9767107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9767523Z return func(*args, **kwargs) 2025-12-04T09:40:08.9767931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9768373Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9768826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9769268Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9769713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:08.9770261Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:08.9770734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:08.9771214Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:08.9771635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:08.9772017Z return self.act(input) 2025-12-04T09:40:08.9772147Z 2025-12-04T09:40:08.9772264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9772702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9773049Z res = mod(**inputs) 2025-12-04T09:40:08.9773442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9773875Z outputs = self.bert( 2025-12-04T09:40:08.9774265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9774690Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9775107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9775524Z layer_outputs = layer_module( 2025-12-04T09:40:08.9775894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9776281Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9776696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9777099Z return func(*args, **kwargs) 2025-12-04T09:40:08.9777493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9777915Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9778349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9778768Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9779207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:08.9779712Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:08.9780184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:08.9780607Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9780763Z 2025-12-04T09:40:08.9780877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9781268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9781606Z res = mod(**inputs) 2025-12-04T09:40:08.9781989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9782391Z outputs = self.bert( 2025-12-04T09:40:08.9782776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9783181Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9783589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9784000Z layer_outputs = layer_module( 2025-12-04T09:40:08.9784374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9784758Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9785218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9785596Z return func(*args, **kwargs) 2025-12-04T09:40:08.9785964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9786364Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9786753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9787129Z return func(*args, **kwargs) 2025-12-04T09:40:08.9787497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9787920Z self_outputs = self.self( 2025-12-04T09:40:08.9788291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9788664Z return func(*args, **kwargs) 2025-12-04T09:40:08.9789038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:08.9789571Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:08.9789839Z 2025-12-04T09:40:08.9789955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9790319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9790650Z res = mod(**inputs) 2025-12-04T09:40:08.9791015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9791403Z outputs = self.bert( 2025-12-04T09:40:08.9791760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9792150Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9792541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9792942Z layer_outputs = layer_module( 2025-12-04T09:40:08.9793318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9793714Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9794100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9794474Z return func(*args, **kwargs) 2025-12-04T09:40:08.9794854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9795254Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9795636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9796016Z return func(*args, **kwargs) 2025-12-04T09:40:08.9796390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9796773Z self_outputs = self.self( 2025-12-04T09:40:08.9797136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9797526Z return func(*args, **kwargs) 2025-12-04T09:40:08.9797923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:08.9798336Z self.key(current_states) 2025-12-04T09:40:08.9798464Z 2025-12-04T09:40:08.9798576Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9798961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9799307Z res = mod(**inputs) 2025-12-04T09:40:08.9799727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9800137Z outputs = self.bert( 2025-12-04T09:40:08.9800595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9801026Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9801435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9801860Z layer_outputs = layer_module( 2025-12-04T09:40:08.9802293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9802685Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9803113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9803522Z return func(*args, **kwargs) 2025-12-04T09:40:08.9803921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9804334Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9804746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9805145Z return func(*args, **kwargs) 2025-12-04T09:40:08.9805533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9805943Z self_outputs = self.self( 2025-12-04T09:40:08.9806335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9806739Z return func(*args, **kwargs) 2025-12-04T09:40:08.9807131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:08.9807538Z self.value(current_states) 2025-12-04T09:40:08.9807665Z 2025-12-04T09:40:08.9807760Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9808023Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9808403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9808746Z res = mod(**inputs) 2025-12-04T09:40:08.9809129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9809537Z outputs = self.bert( 2025-12-04T09:40:08.9809923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9810349Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9810747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9811147Z layer_outputs = layer_module( 2025-12-04T09:40:08.9811522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9811919Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9812320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9812720Z return func(*args, **kwargs) 2025-12-04T09:40:08.9813114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9813540Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9813945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9814346Z return func(*args, **kwargs) 2025-12-04T09:40:08.9814777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9815179Z self_outputs = self.self( 2025-12-04T09:40:08.9815579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9815977Z return func(*args, **kwargs) 2025-12-04T09:40:08.9816354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:08.9816795Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:08.9816992Z 2025-12-04T09:40:08.9817143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9817547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9817904Z res = mod(**inputs) 2025-12-04T09:40:08.9818290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9818703Z outputs = self.bert( 2025-12-04T09:40:08.9819102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9819492Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9819881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9820329Z layer_outputs = layer_module( 2025-12-04T09:40:08.9820685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9821054Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9821444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9821828Z return func(*args, **kwargs) 2025-12-04T09:40:08.9822228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9822717Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9823111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9823491Z return func(*args, **kwargs) 2025-12-04T09:40:08.9823863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:08.9824310Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:08.9824757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:08.9825215Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9825365Z 2025-12-04T09:40:08.9825480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9825880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9826229Z res = mod(**inputs) 2025-12-04T09:40:08.9826605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9827022Z outputs = self.bert( 2025-12-04T09:40:08.9827409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9827824Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9828203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9828595Z layer_outputs = layer_module( 2025-12-04T09:40:08.9828959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9829351Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9829797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9830196Z return func(*args, **kwargs) 2025-12-04T09:40:08.9830591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9831018Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9831461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9831864Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9832355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:08.9832852Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:08.9833311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:08.9833735Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9833883Z 2025-12-04T09:40:08.9833998Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9834390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9834736Z res = mod(**inputs) 2025-12-04T09:40:08.9835117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9835523Z outputs = self.bert( 2025-12-04T09:40:08.9835910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9836324Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9836730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9837142Z layer_outputs = layer_module( 2025-12-04T09:40:08.9837521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9837921Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9838325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9838727Z return func(*args, **kwargs) 2025-12-04T09:40:08.9839122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9839543Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9839970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9840395Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9840917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:08.9841406Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:08.9841867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:08.9842335Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:08.9842777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:08.9843141Z return self.act(input) 2025-12-04T09:40:08.9843274Z 2025-12-04T09:40:08.9843391Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9843784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9844133Z res = mod(**inputs) 2025-12-04T09:40:08.9844510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9844958Z outputs = self.bert( 2025-12-04T09:40:08.9845346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9845753Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9846157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9846562Z layer_outputs = layer_module( 2025-12-04T09:40:08.9846936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9847537Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9847948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9848345Z return func(*args, **kwargs) 2025-12-04T09:40:08.9848742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9849168Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9849605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9850030Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9850462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:08.9850988Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:08.9851465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:08.9851907Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9852058Z 2025-12-04T09:40:08.9852171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9852567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9852916Z res = mod(**inputs) 2025-12-04T09:40:08.9853292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9853723Z outputs = self.bert( 2025-12-04T09:40:08.9854109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9854521Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9854925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9855336Z layer_outputs = layer_module( 2025-12-04T09:40:08.9855717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9856093Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9856481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9856873Z return func(*args, **kwargs) 2025-12-04T09:40:08.9857269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9857671Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9858060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9858439Z return func(*args, **kwargs) 2025-12-04T09:40:08.9858842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9859252Z self_outputs = self.self( 2025-12-04T09:40:08.9859649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9860138Z return func(*args, **kwargs) 2025-12-04T09:40:08.9860541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:08.9861107Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:08.9861398Z 2025-12-04T09:40:08.9861512Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9861915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9862266Z res = mod(**inputs) 2025-12-04T09:40:08.9862720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9863143Z outputs = self.bert( 2025-12-04T09:40:08.9863541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9863959Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9864360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9864777Z layer_outputs = layer_module( 2025-12-04T09:40:08.9865144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9865560Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9865995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9866402Z return func(*args, **kwargs) 2025-12-04T09:40:08.9866797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9867221Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9867635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9868034Z return func(*args, **kwargs) 2025-12-04T09:40:08.9868484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9868869Z self_outputs = self.self( 2025-12-04T09:40:08.9869235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9869606Z return func(*args, **kwargs) 2025-12-04T09:40:08.9869991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:08.9870401Z self.key(current_states) 2025-12-04T09:40:08.9870526Z 2025-12-04T09:40:08.9870638Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9871024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9871368Z res = mod(**inputs) 2025-12-04T09:40:08.9871755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9872158Z outputs = self.bert( 2025-12-04T09:40:08.9872545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9872971Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9873373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9873782Z layer_outputs = layer_module( 2025-12-04T09:40:08.9874160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9874551Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9874950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9875394Z return func(*args, **kwargs) 2025-12-04T09:40:08.9875791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9876222Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9876624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9877022Z return func(*args, **kwargs) 2025-12-04T09:40:08.9877415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9877848Z self_outputs = self.self( 2025-12-04T09:40:08.9878234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9878636Z return func(*args, **kwargs) 2025-12-04T09:40:08.9879033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:08.9879431Z self.value(current_states) 2025-12-04T09:40:08.9879563Z 2025-12-04T09:40:08.9879653Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9879913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9880293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9880712Z res = mod(**inputs) 2025-12-04T09:40:08.9881102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9881526Z outputs = self.bert( 2025-12-04T09:40:08.9881931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9882349Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9882757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9883169Z layer_outputs = layer_module( 2025-12-04T09:40:08.9883540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9883934Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9884344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9884736Z return func(*args, **kwargs) 2025-12-04T09:40:08.9885133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9885561Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9885971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9886366Z return func(*args, **kwargs) 2025-12-04T09:40:08.9886762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9887168Z self_outputs = self.self( 2025-12-04T09:40:08.9887549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9887960Z return func(*args, **kwargs) 2025-12-04T09:40:08.9888333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:08.9888782Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:08.9888976Z 2025-12-04T09:40:08.9889083Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9889458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9889789Z res = mod(**inputs) 2025-12-04T09:40:08.9890193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9890576Z outputs = self.bert( 2025-12-04T09:40:08.9890943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9891333Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9891708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9892094Z layer_outputs = layer_module( 2025-12-04T09:40:08.9892454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9892876Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9893259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9893637Z return func(*args, **kwargs) 2025-12-04T09:40:08.9894049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9894462Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9894876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9895274Z return func(*args, **kwargs) 2025-12-04T09:40:08.9895667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:08.9896105Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:08.9896547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:08.9896947Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9897089Z 2025-12-04T09:40:08.9897203Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9897568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9897898Z res = mod(**inputs) 2025-12-04T09:40:08.9898263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9898640Z outputs = self.bert( 2025-12-04T09:40:08.9899005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9899395Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9899775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9900154Z layer_outputs = layer_module( 2025-12-04T09:40:08.9900508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9900875Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9901255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9901639Z return func(*args, **kwargs) 2025-12-04T09:40:08.9902034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9902452Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9902860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9903266Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9903682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:08.9904142Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:08.9904603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:08.9905005Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9905145Z 2025-12-04T09:40:08.9905260Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9905622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9905957Z res = mod(**inputs) 2025-12-04T09:40:08.9906323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9906704Z outputs = self.bert( 2025-12-04T09:40:08.9907098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9907489Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9907873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9908255Z layer_outputs = layer_module( 2025-12-04T09:40:08.9908614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9908985Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9909374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9909745Z return func(*args, **kwargs) 2025-12-04T09:40:08.9910121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9910520Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9910934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9911329Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9911750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:08.9912215Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:08.9912642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:08.9913070Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:08.9913459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:08.9913809Z return self.act(input) 2025-12-04T09:40:08.9913927Z 2025-12-04T09:40:08.9914035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9914404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9914735Z res = mod(**inputs) 2025-12-04T09:40:08.9915090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9915479Z outputs = self.bert( 2025-12-04T09:40:08.9915845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9916236Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9916614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9917000Z layer_outputs = layer_module( 2025-12-04T09:40:08.9917357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9917730Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9918113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9918494Z return func(*args, **kwargs) 2025-12-04T09:40:08.9918918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9919332Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9919760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9920179Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9920694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:08.9921198Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:08.9922471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:08.9922905Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9923053Z 2025-12-04T09:40:08.9923175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9923572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9923925Z res = mod(**inputs) 2025-12-04T09:40:08.9924310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9924719Z outputs = self.bert( 2025-12-04T09:40:08.9925116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9925542Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9925955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9926369Z layer_outputs = layer_module( 2025-12-04T09:40:08.9926743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9927134Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9927547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9927956Z return func(*args, **kwargs) 2025-12-04T09:40:08.9928361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9928788Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9929199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9929603Z return func(*args, **kwargs) 2025-12-04T09:40:08.9929999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9930408Z self_outputs = self.self( 2025-12-04T09:40:08.9930800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9931197Z return func(*args, **kwargs) 2025-12-04T09:40:08.9931598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:08.9932171Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:08.9932478Z 2025-12-04T09:40:08.9932602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9932989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9933335Z res = mod(**inputs) 2025-12-04T09:40:08.9933712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9934115Z outputs = self.bert( 2025-12-04T09:40:08.9934498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9934950Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9935356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9935771Z layer_outputs = layer_module( 2025-12-04T09:40:08.9936147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9936526Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9936931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9937402Z return func(*args, **kwargs) 2025-12-04T09:40:08.9937785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9938199Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9938612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9939011Z return func(*args, **kwargs) 2025-12-04T09:40:08.9939394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9939811Z self_outputs = self.self( 2025-12-04T09:40:08.9940209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9940624Z return func(*args, **kwargs) 2025-12-04T09:40:08.9941023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:08.9941451Z self.key(current_states) 2025-12-04T09:40:08.9941577Z 2025-12-04T09:40:08.9941696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9942080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9942432Z res = mod(**inputs) 2025-12-04T09:40:08.9942825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9943242Z outputs = self.bert( 2025-12-04T09:40:08.9943630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9944054Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9944469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9944883Z layer_outputs = layer_module( 2025-12-04T09:40:08.9945272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9945676Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9946098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9946499Z return func(*args, **kwargs) 2025-12-04T09:40:08.9946904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9947460Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9947894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9948298Z return func(*args, **kwargs) 2025-12-04T09:40:08.9948709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9949137Z self_outputs = self.self( 2025-12-04T09:40:08.9949535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9949953Z return func(*args, **kwargs) 2025-12-04T09:40:08.9950453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:08.9950871Z self.value(current_states) 2025-12-04T09:40:08.9951003Z 2025-12-04T09:40:08.9951092Z cudagraph partition due to non gpu ops 2025-12-04T09:40:08.9951359Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9951758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9952111Z res = mod(**inputs) 2025-12-04T09:40:08.9952507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9952988Z outputs = self.bert( 2025-12-04T09:40:08.9953384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9953808Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9954211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9954619Z layer_outputs = layer_module( 2025-12-04T09:40:08.9954986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9955377Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9955782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9956188Z return func(*args, **kwargs) 2025-12-04T09:40:08.9956572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9956996Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9957409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9957816Z return func(*args, **kwargs) 2025-12-04T09:40:08.9958202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:08.9958606Z self_outputs = self.self( 2025-12-04T09:40:08.9958994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9959445Z return func(*args, **kwargs) 2025-12-04T09:40:08.9959849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:08.9960350Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:08.9960628Z 2025-12-04T09:40:08.9960756Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9961141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9961488Z res = mod(**inputs) 2025-12-04T09:40:08.9961880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9962281Z outputs = self.bert( 2025-12-04T09:40:08.9962674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9963087Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9963495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9963894Z layer_outputs = layer_module( 2025-12-04T09:40:08.9964268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9964645Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9965026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9965442Z return func(*args, **kwargs) 2025-12-04T09:40:08.9965818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:08.9966221Z self_attention_outputs = self.attention( 2025-12-04T09:40:08.9966603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9966982Z return func(*args, **kwargs) 2025-12-04T09:40:08.9967357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:08.9967838Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:08.9968270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:08.9968669Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9968810Z 2025-12-04T09:40:08.9968936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9969295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9969644Z res = mod(**inputs) 2025-12-04T09:40:08.9970026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9970434Z outputs = self.bert( 2025-12-04T09:40:08.9970814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9971229Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9971635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9972054Z layer_outputs = layer_module( 2025-12-04T09:40:08.9972420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9972816Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9973225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9973628Z return func(*args, **kwargs) 2025-12-04T09:40:08.9974025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9974460Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9974897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9975317Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9975758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:08.9976251Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:08.9976701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:08.9977122Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9977275Z 2025-12-04T09:40:08.9977387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9977773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9978115Z res = mod(**inputs) 2025-12-04T09:40:08.9978495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9978901Z outputs = self.bert( 2025-12-04T09:40:08.9979284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9979690Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9980127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9980542Z layer_outputs = layer_module( 2025-12-04T09:40:08.9980912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9981309Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9981725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9982128Z return func(*args, **kwargs) 2025-12-04T09:40:08.9982522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9982985Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9983423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9983840Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9984283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:08.9984759Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:08.9985215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:08.9985672Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:08.9986082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:08.9986464Z return self.act(input) 2025-12-04T09:40:08.9986587Z 2025-12-04T09:40:08.9986704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9987096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9987472Z res = mod(**inputs) 2025-12-04T09:40:08.9987844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9988222Z outputs = self.bert( 2025-12-04T09:40:08.9988590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9988980Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9989365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9989743Z layer_outputs = layer_module( 2025-12-04T09:40:08.9990101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9990475Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9990857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9991261Z return func(*args, **kwargs) 2025-12-04T09:40:08.9991673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:08.9992092Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:08.9992519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:08.9992947Z return forward_fn(*input_tensors) 2025-12-04T09:40:08.9993363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:08.9993841Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:08.9994282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:08.9994680Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:08.9994818Z 2025-12-04T09:40:08.9994968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:08.9995339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:08.9995665Z res = mod(**inputs) 2025-12-04T09:40:08.9996035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:08.9996419Z outputs = self.bert( 2025-12-04T09:40:08.9996778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:08.9997177Z encoder_outputs = self.encoder( 2025-12-04T09:40:08.9997618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:08.9998025Z layer_outputs = layer_module( 2025-12-04T09:40:08.9998406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:08.9998814Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:08.9999232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:08.9999646Z return func(*args, **kwargs) 2025-12-04T09:40:09.0000054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0000564Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0001007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0001428Z return func(*args, **kwargs) 2025-12-04T09:40:09.0001843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0002279Z self_outputs = self.self( 2025-12-04T09:40:09.0002687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0003091Z return func(*args, **kwargs) 2025-12-04T09:40:09.0003491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:09.0004053Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:09.0004334Z 2025-12-04T09:40:09.0004447Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0004839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0005191Z res = mod(**inputs) 2025-12-04T09:40:09.0005578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0005979Z outputs = self.bert( 2025-12-04T09:40:09.0006371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0006786Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0007189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0007602Z layer_outputs = layer_module( 2025-12-04T09:40:09.0007980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0008371Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0008775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0009182Z return func(*args, **kwargs) 2025-12-04T09:40:09.0009589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0009984Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0010429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0010806Z return func(*args, **kwargs) 2025-12-04T09:40:09.0011183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0011563Z self_outputs = self.self( 2025-12-04T09:40:09.0011935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0012327Z return func(*args, **kwargs) 2025-12-04T09:40:09.0012755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:09.0013172Z self.key(current_states) 2025-12-04T09:40:09.0013302Z 2025-12-04T09:40:09.0013414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0013804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0014146Z res = mod(**inputs) 2025-12-04T09:40:09.0014509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0014889Z outputs = self.bert( 2025-12-04T09:40:09.0015250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0015630Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0016019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0016431Z layer_outputs = layer_module( 2025-12-04T09:40:09.0016796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0017181Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0017590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0017986Z return func(*args, **kwargs) 2025-12-04T09:40:09.0018352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0018745Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0019135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0019532Z return func(*args, **kwargs) 2025-12-04T09:40:09.0019924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0020311Z self_outputs = self.self( 2025-12-04T09:40:09.0020680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0021046Z return func(*args, **kwargs) 2025-12-04T09:40:09.0021420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:09.0021826Z self.value(current_states) 2025-12-04T09:40:09.0021955Z 2025-12-04T09:40:09.0022051Z cudagraph partition due to non gpu ops 2025-12-04T09:40:09.0022305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0022692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0023036Z res = mod(**inputs) 2025-12-04T09:40:09.0023413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0023816Z outputs = self.bert( 2025-12-04T09:40:09.0024204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0024589Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0025002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0025397Z layer_outputs = layer_module( 2025-12-04T09:40:09.0025772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0026155Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0026567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0026968Z return func(*args, **kwargs) 2025-12-04T09:40:09.0027398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0027812Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0028222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0028625Z return func(*args, **kwargs) 2025-12-04T09:40:09.0029013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0029417Z self_outputs = self.self( 2025-12-04T09:40:09.0029812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0030210Z return func(*args, **kwargs) 2025-12-04T09:40:09.0030597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:09.0031071Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:09.0031271Z 2025-12-04T09:40:09.0031392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0031779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0032111Z res = mod(**inputs) 2025-12-04T09:40:09.0032505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0032931Z outputs = self.bert( 2025-12-04T09:40:09.0033308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0033725Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0034126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0034531Z layer_outputs = layer_module( 2025-12-04T09:40:09.0034902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0035292Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0035710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0036124Z return func(*args, **kwargs) 2025-12-04T09:40:09.0036532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0036978Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0037403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0037813Z return func(*args, **kwargs) 2025-12-04T09:40:09.0038219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:09.0038707Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:09.0039184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:09.0039625Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0039786Z 2025-12-04T09:40:09.0039943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0040341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0040777Z res = mod(**inputs) 2025-12-04T09:40:09.0041201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0041279Z outputs = self.bert( 2025-12-04T09:40:09.0041564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0041691Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0041982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0042061Z layer_outputs = layer_module( 2025-12-04T09:40:09.0042305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0042403Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0042663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0042748Z return func(*args, **kwargs) 2025-12-04T09:40:09.0043013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0043111Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0043405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0043492Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0043791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0043929Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0044197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:09.0044294Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0044299Z 2025-12-04T09:40:09.0044411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0044620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0044699Z res = mod(**inputs) 2025-12-04T09:40:09.0044968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0045051Z outputs = self.bert( 2025-12-04T09:40:09.0045316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0045395Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0045670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0045746Z layer_outputs = layer_module( 2025-12-04T09:40:09.0045984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0046078Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0046334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0046415Z return func(*args, **kwargs) 2025-12-04T09:40:09.0046678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0046772Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0047217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0047311Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0047698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0047833Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0048104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:09.0048234Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:09.0048466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:09.0048610Z return self.act(input) 2025-12-04T09:40:09.0048615Z 2025-12-04T09:40:09.0048737Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0048951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0049029Z res = mod(**inputs) 2025-12-04T09:40:09.0049300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0049373Z outputs = self.bert( 2025-12-04T09:40:09.0049651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0049731Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0049997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0050082Z layer_outputs = layer_module( 2025-12-04T09:40:09.0050322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0050415Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0050673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0050751Z return func(*args, **kwargs) 2025-12-04T09:40:09.0051024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0051111Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0051398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0051479Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0051773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:09.0051926Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:09.0052191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:09.0052279Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0052290Z 2025-12-04T09:40:09.0052405Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0052619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0052694Z res = mod(**inputs) 2025-12-04T09:40:09.0052964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0053036Z outputs = self.bert( 2025-12-04T09:40:09.0053310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0053392Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0053666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0053741Z layer_outputs = layer_module( 2025-12-04T09:40:09.0054013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0054109Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0054371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0054446Z return func(*args, **kwargs) 2025-12-04T09:40:09.0054717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0054806Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0055072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0055186Z return func(*args, **kwargs) 2025-12-04T09:40:09.0055456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0055540Z self_outputs = self.self( 2025-12-04T09:40:09.0055811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0055884Z return func(*args, **kwargs) 2025-12-04T09:40:09.0056160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:09.0056384Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:09.0056388Z 2025-12-04T09:40:09.0056509Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0056726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0056797Z res = mod(**inputs) 2025-12-04T09:40:09.0057082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0057153Z outputs = self.bert( 2025-12-04T09:40:09.0057436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0057515Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0057785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0057869Z layer_outputs = layer_module( 2025-12-04T09:40:09.0058114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0058199Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0058474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0058550Z return func(*args, **kwargs) 2025-12-04T09:40:09.0058826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0058913Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0059178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0059260Z return func(*args, **kwargs) 2025-12-04T09:40:09.0059528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0059612Z self_outputs = self.self( 2025-12-04T09:40:09.0059875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0059948Z return func(*args, **kwargs) 2025-12-04T09:40:09.0060226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:09.0060303Z self.key(current_states) 2025-12-04T09:40:09.0060307Z 2025-12-04T09:40:09.0060419Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0060679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0060749Z res = mod(**inputs) 2025-12-04T09:40:09.0061022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0061094Z outputs = self.bert( 2025-12-04T09:40:09.0061361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0061448Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0061710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0061827Z layer_outputs = layer_module( 2025-12-04T09:40:09.0062077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0062161Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0062433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0062508Z return func(*args, **kwargs) 2025-12-04T09:40:09.0062787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0062883Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0063147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0063228Z return func(*args, **kwargs) 2025-12-04T09:40:09.0063501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0063577Z self_outputs = self.self( 2025-12-04T09:40:09.0063851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0063927Z return func(*args, **kwargs) 2025-12-04T09:40:09.0064183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:09.0064266Z self.value(current_states) 2025-12-04T09:40:09.0064269Z 2025-12-04T09:40:09.0064353Z cudagraph partition due to non gpu ops 2025-12-04T09:40:09.0064474Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0064693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0064763Z res = mod(**inputs) 2025-12-04T09:40:09.0065058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0065127Z outputs = self.bert( 2025-12-04T09:40:09.0065385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0065470Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0065726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0065803Z layer_outputs = layer_module( 2025-12-04T09:40:09.0066031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0066111Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0066376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0066453Z return func(*args, **kwargs) 2025-12-04T09:40:09.0066722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0066820Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0067117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0067197Z return func(*args, **kwargs) 2025-12-04T09:40:09.0067462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0067535Z self_outputs = self.self( 2025-12-04T09:40:09.0067804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0067876Z return func(*args, **kwargs) 2025-12-04T09:40:09.0068148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:09.0068339Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:09.0068343Z 2025-12-04T09:40:09.0068454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0068669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0068741Z res = mod(**inputs) 2025-12-04T09:40:09.0069007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0069081Z outputs = self.bert( 2025-12-04T09:40:09.0069333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0069413Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0069660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0069733Z layer_outputs = layer_module( 2025-12-04T09:40:09.0069964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0070044Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0070297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0070367Z return func(*args, **kwargs) 2025-12-04T09:40:09.0070616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0070709Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0070966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0071038Z return func(*args, **kwargs) 2025-12-04T09:40:09.0071311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:09.0071451Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:09.0071719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:09.0071810Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0071816Z 2025-12-04T09:40:09.0071928Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0072147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0072218Z res = mod(**inputs) 2025-12-04T09:40:09.0072484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0072564Z outputs = self.bert( 2025-12-04T09:40:09.0072829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0072920Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0073183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0073267Z layer_outputs = layer_module( 2025-12-04T09:40:09.0073533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0073617Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0073869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0073938Z return func(*args, **kwargs) 2025-12-04T09:40:09.0074189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0074284Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0074555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0074684Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0074991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0075122Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0075394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:09.0075483Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0075486Z 2025-12-04T09:40:09.0075596Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0075818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0075887Z res = mod(**inputs) 2025-12-04T09:40:09.0076160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0076236Z outputs = self.bert( 2025-12-04T09:40:09.0076498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0076585Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0076852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0076928Z layer_outputs = layer_module( 2025-12-04T09:40:09.0077173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0077258Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0077523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0077596Z return func(*args, **kwargs) 2025-12-04T09:40:09.0077861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0077959Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0078239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0078325Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0078629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0078756Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0079027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:09.0079149Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:09.0079378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:09.0079467Z return self.act(input) 2025-12-04T09:40:09.0079470Z 2025-12-04T09:40:09.0079584Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0079806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0079910Z res = mod(**inputs) 2025-12-04T09:40:09.0080179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0080260Z outputs = self.bert( 2025-12-04T09:40:09.0080599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0080687Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0080960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0081035Z layer_outputs = layer_module( 2025-12-04T09:40:09.0081327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0081414Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0081677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0081767Z return func(*args, **kwargs) 2025-12-04T09:40:09.0082039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0082139Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0082429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0082513Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0082826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:09.0082984Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:09.0083249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:09.0083345Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0083352Z 2025-12-04T09:40:09.0083462Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0083678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0083748Z res = mod(**inputs) 2025-12-04T09:40:09.0084012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0084091Z outputs = self.bert( 2025-12-04T09:40:09.0084355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0084442Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0084703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0084778Z layer_outputs = layer_module( 2025-12-04T09:40:09.0085023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0085109Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0085367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0085447Z return func(*args, **kwargs) 2025-12-04T09:40:09.0085720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0085814Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0086080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0086157Z return func(*args, **kwargs) 2025-12-04T09:40:09.0086429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0086504Z self_outputs = self.self( 2025-12-04T09:40:09.0086803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0086887Z return func(*args, **kwargs) 2025-12-04T09:40:09.0087166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:09.0087401Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:09.0087405Z 2025-12-04T09:40:09.0087519Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0087768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0087848Z res = mod(**inputs) 2025-12-04T09:40:09.0088126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0088206Z outputs = self.bert( 2025-12-04T09:40:09.0088486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0088564Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0088840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0088918Z layer_outputs = layer_module( 2025-12-04T09:40:09.0089157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0089250Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0089512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0089593Z return func(*args, **kwargs) 2025-12-04T09:40:09.0089868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0089959Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0090224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0090296Z return func(*args, **kwargs) 2025-12-04T09:40:09.0090579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0090656Z self_outputs = self.self( 2025-12-04T09:40:09.0090924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0091006Z return func(*args, **kwargs) 2025-12-04T09:40:09.0091281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:09.0091356Z self.key(current_states) 2025-12-04T09:40:09.0091359Z 2025-12-04T09:40:09.0091478Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0091705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0091781Z res = mod(**inputs) 2025-12-04T09:40:09.0092060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0092130Z outputs = self.bert( 2025-12-04T09:40:09.0092410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0092488Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0092762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0092847Z layer_outputs = layer_module( 2025-12-04T09:40:09.0093086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0093209Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0093482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0093555Z return func(*args, **kwargs) 2025-12-04T09:40:09.0093832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0093920Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0094202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0094324Z return func(*args, **kwargs) 2025-12-04T09:40:09.0094595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0094677Z self_outputs = self.self( 2025-12-04T09:40:09.0094957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0095029Z return func(*args, **kwargs) 2025-12-04T09:40:09.0095321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:09.0095398Z self.value(current_states) 2025-12-04T09:40:09.0095402Z 2025-12-04T09:40:09.0095498Z cudagraph partition due to non gpu ops 2025-12-04T09:40:09.0095610Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0095839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0095915Z res = mod(**inputs) 2025-12-04T09:40:09.0096197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0096268Z outputs = self.bert( 2025-12-04T09:40:09.0096547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0096627Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0096907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0096983Z layer_outputs = layer_module( 2025-12-04T09:40:09.0097228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0097320Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0097585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0097659Z return func(*args, **kwargs) 2025-12-04T09:40:09.0097937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0098025Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0098298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0098372Z return func(*args, **kwargs) 2025-12-04T09:40:09.0098648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0098731Z self_outputs = self.self( 2025-12-04T09:40:09.0099003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0099082Z return func(*args, **kwargs) 2025-12-04T09:40:09.0099383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:09.0099529Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:09.0099533Z 2025-12-04T09:40:09.0099653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0099943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0100013Z res = mod(**inputs) 2025-12-04T09:40:09.0100287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0100357Z outputs = self.bert( 2025-12-04T09:40:09.0100630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0100709Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0100969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0101088Z layer_outputs = layer_module( 2025-12-04T09:40:09.0101325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0101417Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0101678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0101752Z return func(*args, **kwargs) 2025-12-04T09:40:09.0102023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0102109Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0102367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0102448Z return func(*args, **kwargs) 2025-12-04T09:40:09.0102719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:09.0102873Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:09.0103143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:09.0103239Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0103243Z 2025-12-04T09:40:09.0103364Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0103581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0103654Z res = mod(**inputs) 2025-12-04T09:40:09.0103934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0104009Z outputs = self.bert( 2025-12-04T09:40:09.0104288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0104373Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0104646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0104731Z layer_outputs = layer_module( 2025-12-04T09:40:09.0104972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0105063Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0105321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0105393Z return func(*args, **kwargs) 2025-12-04T09:40:09.0105661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0105752Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0106035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0106123Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0106418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0106588Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0106855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:09.0106946Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0106950Z 2025-12-04T09:40:09.0107074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0107292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0107368Z res = mod(**inputs) 2025-12-04T09:40:09.0107647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0107767Z outputs = self.bert( 2025-12-04T09:40:09.0108046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0108124Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0108392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0108475Z layer_outputs = layer_module( 2025-12-04T09:40:09.0108713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0108802Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0109059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0109134Z return func(*args, **kwargs) 2025-12-04T09:40:09.0109408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0109500Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0109789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0109879Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0110184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0110324Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0110596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:09.0110722Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:09.0110967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:09.0111047Z return self.act(input) 2025-12-04T09:40:09.0111051Z 2025-12-04T09:40:09.0111171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0111390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0111464Z res = mod(**inputs) 2025-12-04T09:40:09.0111750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0111823Z outputs = self.bert( 2025-12-04T09:40:09.0112103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0112187Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0112452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0112539Z layer_outputs = layer_module( 2025-12-04T09:40:09.0112775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0112860Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0113168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0113246Z return func(*args, **kwargs) 2025-12-04T09:40:09.0113524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0113615Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0113899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0113992Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0114294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:09.0114477Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:09.0114760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:09.0114855Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0114859Z 2025-12-04T09:40:09.0114980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0115196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0115267Z res = mod(**inputs) 2025-12-04T09:40:09.0115549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0115621Z outputs = self.bert( 2025-12-04T09:40:09.0115901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0115983Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0116256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0116338Z layer_outputs = layer_module( 2025-12-04T09:40:09.0116583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0116670Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0116944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0117019Z return func(*args, **kwargs) 2025-12-04T09:40:09.0117295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0117386Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0117654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0117736Z return func(*args, **kwargs) 2025-12-04T09:40:09.0118006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0118085Z self_outputs = self.self( 2025-12-04T09:40:09.0118358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0118433Z return func(*args, **kwargs) 2025-12-04T09:40:09.0118713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:09.0118942Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:09.0118946Z 2025-12-04T09:40:09.0119058Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0119287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0119357Z res = mod(**inputs) 2025-12-04T09:40:09.0119637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0119744Z outputs = self.bert( 2025-12-04T09:40:09.0120025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0120113Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0120388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0120468Z layer_outputs = layer_module( 2025-12-04T09:40:09.0120823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0120914Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0121238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0121316Z return func(*args, **kwargs) 2025-12-04T09:40:09.0121591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0121691Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0121956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0122040Z return func(*args, **kwargs) 2025-12-04T09:40:09.0122313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0122390Z self_outputs = self.self( 2025-12-04T09:40:09.0122664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0122742Z return func(*args, **kwargs) 2025-12-04T09:40:09.0123012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:09.0123099Z self.key(current_states) 2025-12-04T09:40:09.0123103Z 2025-12-04T09:40:09.0123220Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0123449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0123520Z res = mod(**inputs) 2025-12-04T09:40:09.0123794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0123876Z outputs = self.bert( 2025-12-04T09:40:09.0124150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0124231Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0124513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0124590Z layer_outputs = layer_module( 2025-12-04T09:40:09.0124840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0124929Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0125195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0125278Z return func(*args, **kwargs) 2025-12-04T09:40:09.0125549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0125640Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0125917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0125994Z return func(*args, **kwargs) 2025-12-04T09:40:09.0126274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0126349Z self_outputs = self.self( 2025-12-04T09:40:09.0126661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0126747Z return func(*args, **kwargs) 2025-12-04T09:40:09.0127020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:09.0127106Z self.value(current_states) 2025-12-04T09:40:09.0127111Z 2025-12-04T09:40:09.0127204Z cudagraph partition due to non gpu ops 2025-12-04T09:40:09.0127318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0127542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0127646Z res = mod(**inputs) 2025-12-04T09:40:09.0127944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0128027Z outputs = self.bert( 2025-12-04T09:40:09.0128327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0128414Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0128708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0128785Z layer_outputs = layer_module( 2025-12-04T09:40:09.0129044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0129129Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0129413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0129496Z return func(*args, **kwargs) 2025-12-04T09:40:09.0129789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0129898Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0130180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0130252Z return func(*args, **kwargs) 2025-12-04T09:40:09.0130537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0130613Z self_outputs = self.self( 2025-12-04T09:40:09.0130907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0130981Z return func(*args, **kwargs) 2025-12-04T09:40:09.0131279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:09.0131438Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:09.0131442Z 2025-12-04T09:40:09.0131552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0131780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0131859Z res = mod(**inputs) 2025-12-04T09:40:09.0132147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0132223Z outputs = self.bert( 2025-12-04T09:40:09.0132511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0132590Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0132872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0132949Z layer_outputs = layer_module( 2025-12-04T09:40:09.0133196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0133288Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0133603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0133685Z return func(*args, **kwargs) 2025-12-04T09:40:09.0133964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0134053Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0134330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0134403Z return func(*args, **kwargs) 2025-12-04T09:40:09.0134723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:09.0134861Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:09.0135132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:09.0135232Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0135236Z 2025-12-04T09:40:09.0135347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0135558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0135634Z res = mod(**inputs) 2025-12-04T09:40:09.0135911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0135988Z outputs = self.bert( 2025-12-04T09:40:09.0136265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0136346Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0136615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0136689Z layer_outputs = layer_module( 2025-12-04T09:40:09.0136937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0137021Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0137278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0137358Z return func(*args, **kwargs) 2025-12-04T09:40:09.0137623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0137712Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0138002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0138084Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0138389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0138516Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0138780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:09.0138876Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0138879Z 2025-12-04T09:40:09.0138989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0139211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0139279Z res = mod(**inputs) 2025-12-04T09:40:09.0139549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0139627Z outputs = self.bert( 2025-12-04T09:40:09.0139893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0140006Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0140278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0140355Z layer_outputs = layer_module( 2025-12-04T09:40:09.0140599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0140682Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0140940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0141056Z return func(*args, **kwargs) 2025-12-04T09:40:09.0141318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0141407Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0141695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0141780Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0142090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0142221Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0142492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:09.0142627Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:09.0142865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:09.0142950Z return self.act(input) 2025-12-04T09:40:09.0142954Z 2025-12-04T09:40:09.0143066Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0143291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0143369Z res = mod(**inputs) 2025-12-04T09:40:09.0143635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0143706Z outputs = self.bert( 2025-12-04T09:40:09.0143982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0144061Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0144332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0144411Z layer_outputs = layer_module( 2025-12-04T09:40:09.0144656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0144748Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0145010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0145092Z return func(*args, **kwargs) 2025-12-04T09:40:09.0145354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0145444Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0145732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0145812Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0146109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:09.0146259Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:09.0146522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:09.0146654Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0146658Z 2025-12-04T09:40:09.0146770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0146984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0147182Z res = mod(**inputs) 2025-12-04T09:40:09.0147459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0147540Z outputs = self.bert( 2025-12-04T09:40:09.0147808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0147989Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0148263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0148338Z layer_outputs = layer_module( 2025-12-04T09:40:09.0148580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0148674Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0148936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0149017Z return func(*args, **kwargs) 2025-12-04T09:40:09.0149282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0149371Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0149646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0149719Z return func(*args, **kwargs) 2025-12-04T09:40:09.0149984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0150071Z self_outputs = self.self( 2025-12-04T09:40:09.0150333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0150412Z return func(*args, **kwargs) 2025-12-04T09:40:09.0150679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:09.0150903Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:09.0150907Z 2025-12-04T09:40:09.0151029Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0151244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0151322Z res = mod(**inputs) 2025-12-04T09:40:09.0151591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0151665Z outputs = self.bert( 2025-12-04T09:40:09.0151939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0152021Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0152287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0152370Z layer_outputs = layer_module( 2025-12-04T09:40:09.0152611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0152707Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0152969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0153044Z return func(*args, **kwargs) 2025-12-04T09:40:09.0153538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0153636Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0153896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0153978Z return func(*args, **kwargs) 2025-12-04T09:40:09.0154241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0154326Z self_outputs = self.self( 2025-12-04T09:40:09.0154583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0154693Z return func(*args, **kwargs) 2025-12-04T09:40:09.0154976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:09.0155052Z self.key(current_states) 2025-12-04T09:40:09.0155056Z 2025-12-04T09:40:09.0155181Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0155399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0155469Z res = mod(**inputs) 2025-12-04T09:40:09.0155751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0155823Z outputs = self.bert( 2025-12-04T09:40:09.0156097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0156187Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0156461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0156546Z layer_outputs = layer_module( 2025-12-04T09:40:09.0156792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0156877Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0157152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0157224Z return func(*args, **kwargs) 2025-12-04T09:40:09.0157492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0157587Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0157853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0157934Z return func(*args, **kwargs) 2025-12-04T09:40:09.0158207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0158280Z self_outputs = self.self( 2025-12-04T09:40:09.0158556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0158627Z return func(*args, **kwargs) 2025-12-04T09:40:09.0158906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:09.0158983Z self.value(current_states) 2025-12-04T09:40:09.0158986Z 2025-12-04T09:40:09.0159075Z cudagraph partition due to non gpu ops 2025-12-04T09:40:09.0159193Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0159411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0159482Z res = mod(**inputs) 2025-12-04T09:40:09.0159765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0159836Z outputs = self.bert( 2025-12-04T09:40:09.0160171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0160255Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0160581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0160675Z layer_outputs = layer_module( 2025-12-04T09:40:09.0160921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0161007Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0161278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0161400Z return func(*args, **kwargs) 2025-12-04T09:40:09.0161692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0161779Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0162043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0162124Z return func(*args, **kwargs) 2025-12-04T09:40:09.0162386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0162468Z self_outputs = self.self( 2025-12-04T09:40:09.0162731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0162802Z return func(*args, **kwargs) 2025-12-04T09:40:09.0163078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:09.0163222Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:09.0163226Z 2025-12-04T09:40:09.0163338Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0163561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0163631Z res = mod(**inputs) 2025-12-04T09:40:09.0163907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0163978Z outputs = self.bert( 2025-12-04T09:40:09.0164247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0164334Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0164601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0164680Z layer_outputs = layer_module( 2025-12-04T09:40:09.0164926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0165009Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0165277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0165350Z return func(*args, **kwargs) 2025-12-04T09:40:09.0165614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0165710Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0165971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0166049Z return func(*args, **kwargs) 2025-12-04T09:40:09.0166330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:09.0166460Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:09.0166754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:09.0166841Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0166845Z 2025-12-04T09:40:09.0166950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0167157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0167222Z res = mod(**inputs) 2025-12-04T09:40:09.0167481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0167548Z outputs = self.bert( 2025-12-04T09:40:09.0167829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0167913Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0168158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0168239Z layer_outputs = layer_module( 2025-12-04T09:40:09.0168465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0168544Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0168792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0168862Z return func(*args, **kwargs) 2025-12-04T09:40:09.0169110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0169207Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0169472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0169557Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0169836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0169957Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0170215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:09.0170302Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0170305Z 2025-12-04T09:40:09.0170417Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0170616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0170685Z res = mod(**inputs) 2025-12-04T09:40:09.0170943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0171011Z outputs = self.bert( 2025-12-04T09:40:09.0171261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0171345Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0171598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0171681Z layer_outputs = layer_module( 2025-12-04T09:40:09.0171916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0171999Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0172270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0172343Z return func(*args, **kwargs) 2025-12-04T09:40:09.0172587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0172680Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0172987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0173074Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0173353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0173474Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0173734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:09.0173850Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:09.0174118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:09.0174190Z return self.act(input) 2025-12-04T09:40:09.0174194Z 2025-12-04T09:40:09.0174301Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0174512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0174582Z res = mod(**inputs) 2025-12-04T09:40:09.0174851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0174932Z outputs = self.bert( 2025-12-04T09:40:09.0175202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0175289Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0175550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0175629Z layer_outputs = layer_module( 2025-12-04T09:40:09.0175874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0175958Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0176218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0176303Z return func(*args, **kwargs) 2025-12-04T09:40:09.0176572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0176664Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0176926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0177003Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0177291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:09.0177426Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:09.0177684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:09.0177767Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0177771Z 2025-12-04T09:40:09.0177874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0178081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0178146Z res = mod(**inputs) 2025-12-04T09:40:09.0178397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0178471Z outputs = self.bert( 2025-12-04T09:40:09.0178722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0178805Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0179050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0179153Z layer_outputs = layer_module( 2025-12-04T09:40:09.0179387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0179468Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0179718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0179789Z return func(*args, **kwargs) 2025-12-04T09:40:09.0180037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0180160Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0180402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0180470Z return func(*args, **kwargs) 2025-12-04T09:40:09.0180726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0180796Z self_outputs = self.self( 2025-12-04T09:40:09.0181045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0181111Z return func(*args, **kwargs) 2025-12-04T09:40:09.0181357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:09.0181574Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:09.0181580Z 2025-12-04T09:40:09.0181685Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0181887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0181951Z res = mod(**inputs) 2025-12-04T09:40:09.0182201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0182275Z outputs = self.bert( 2025-12-04T09:40:09.0182523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0182597Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0182851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0182921Z layer_outputs = layer_module( 2025-12-04T09:40:09.0183155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0183240Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0183480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0183557Z return func(*args, **kwargs) 2025-12-04T09:40:09.0183805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0183888Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0184137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0184206Z return func(*args, **kwargs) 2025-12-04T09:40:09.0184461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0184532Z self_outputs = self.self( 2025-12-04T09:40:09.0184773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0184851Z return func(*args, **kwargs) 2025-12-04T09:40:09.0185097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:09.0185176Z self.key(current_states) 2025-12-04T09:40:09.0185211Z 2025-12-04T09:40:09.0185318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0185519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0185594Z res = mod(**inputs) 2025-12-04T09:40:09.0185847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0185915Z outputs = self.bert( 2025-12-04T09:40:09.0186176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0186287Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0186542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0186613Z layer_outputs = layer_module( 2025-12-04T09:40:09.0186839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0186926Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0187168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0187236Z return func(*args, **kwargs) 2025-12-04T09:40:09.0187493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0187578Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0187827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0187899Z return func(*args, **kwargs) 2025-12-04T09:40:09.0188145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0188223Z self_outputs = self.self( 2025-12-04T09:40:09.0188466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0188540Z return func(*args, **kwargs) 2025-12-04T09:40:09.0188786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:09.0188858Z self.value(current_states) 2025-12-04T09:40:09.0188862Z 2025-12-04T09:40:09.0188953Z cudagraph partition due to non gpu ops 2025-12-04T09:40:09.0189059Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0189263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0189339Z res = mod(**inputs) 2025-12-04T09:40:09.0189592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0189666Z outputs = self.bert( 2025-12-04T09:40:09.0189919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0189996Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0190253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0190324Z layer_outputs = layer_module( 2025-12-04T09:40:09.0190549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0190634Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0190880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0190957Z return func(*args, **kwargs) 2025-12-04T09:40:09.0191202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0191317Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0191571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0191638Z return func(*args, **kwargs) 2025-12-04T09:40:09.0191911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0191986Z self_outputs = self.self( 2025-12-04T09:40:09.0192246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0192361Z return func(*args, **kwargs) 2025-12-04T09:40:09.0192628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:09.0192774Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:09.0192778Z 2025-12-04T09:40:09.0192900Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0193117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0193193Z res = mod(**inputs) 2025-12-04T09:40:09.0193460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0193543Z outputs = self.bert( 2025-12-04T09:40:09.0193806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0193883Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0194140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0194220Z layer_outputs = layer_module( 2025-12-04T09:40:09.0194446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0194534Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0194788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0194860Z return func(*args, **kwargs) 2025-12-04T09:40:09.0195132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0195221Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0195489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0195564Z return func(*args, **kwargs) 2025-12-04T09:40:09.0195832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:09.0195977Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:09.0196244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:09.0196335Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0196347Z 2025-12-04T09:40:09.0196458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0196668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0196745Z res = mod(**inputs) 2025-12-04T09:40:09.0197015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0197090Z outputs = self.bert( 2025-12-04T09:40:09.0197362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0197441Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0197760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0197838Z layer_outputs = layer_module( 2025-12-04T09:40:09.0198076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0198167Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0198423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0198497Z return func(*args, **kwargs) 2025-12-04T09:40:09.0198767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0198899Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0199190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0199274Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0199574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0199713Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0199978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:09.0200065Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0200076Z 2025-12-04T09:40:09.0200188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0200399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0200542Z res = mod(**inputs) 2025-12-04T09:40:09.0200818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0200893Z outputs = self.bert( 2025-12-04T09:40:09.0201180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0201263Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0201543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0201623Z layer_outputs = layer_module( 2025-12-04T09:40:09.0201869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0201965Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0202239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0202316Z return func(*args, **kwargs) 2025-12-04T09:40:09.0202587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0202678Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0202973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0203056Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0203352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0203489Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0203752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:09.0203888Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:09.0204120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:09.0204198Z return self.act(input) 2025-12-04T09:40:09.0204202Z 2025-12-04T09:40:09.0204322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0204605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0204679Z res = mod(**inputs) 2025-12-04T09:40:09.0204953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0205024Z outputs = self.bert( 2025-12-04T09:40:09.0205298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0205378Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0205674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0205759Z layer_outputs = layer_module( 2025-12-04T09:40:09.0206000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0206087Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0206357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0206429Z return func(*args, **kwargs) 2025-12-04T09:40:09.0206699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0206789Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0207068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0207162Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0207457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:09.0207609Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:09.0207875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:09.0207964Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0207968Z 2025-12-04T09:40:09.0208085Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0208296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0208368Z res = mod(**inputs) 2025-12-04T09:40:09.0208639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0208712Z outputs = self.bert( 2025-12-04T09:40:09.0208983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0209060Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0209322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0209405Z layer_outputs = layer_module( 2025-12-04T09:40:09.0209644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0209735Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0209993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0210066Z return func(*args, **kwargs) 2025-12-04T09:40:09.0210331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0210423Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0210678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0210759Z return func(*args, **kwargs) 2025-12-04T09:40:09.0211051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0211135Z self_outputs = self.self( 2025-12-04T09:40:09.0211395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0211475Z return func(*args, **kwargs) 2025-12-04T09:40:09.0211732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:40:09.0211942Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:40:09.0211978Z 2025-12-04T09:40:09.0212091Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0212290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0212353Z res = mod(**inputs) 2025-12-04T09:40:09.0212618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0212687Z outputs = self.bert( 2025-12-04T09:40:09.0212941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0213022Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0213271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0213350Z layer_outputs = layer_module( 2025-12-04T09:40:09.0213574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0213658Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0213907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0213977Z return func(*args, **kwargs) 2025-12-04T09:40:09.0214228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0214320Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0214582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0214658Z return func(*args, **kwargs) 2025-12-04T09:40:09.0214908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0214978Z self_outputs = self.self( 2025-12-04T09:40:09.0215234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0215303Z return func(*args, **kwargs) 2025-12-04T09:40:09.0215561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:40:09.0215634Z self.key(current_states) 2025-12-04T09:40:09.0215637Z 2025-12-04T09:40:09.0215745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0215951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0216018Z res = mod(**inputs) 2025-12-04T09:40:09.0216272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0216349Z outputs = self.bert( 2025-12-04T09:40:09.0216602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0216686Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0216938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0217009Z layer_outputs = layer_module( 2025-12-04T09:40:09.0217275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0217360Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0217613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0217696Z return func(*args, **kwargs) 2025-12-04T09:40:09.0217960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0218056Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0218343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0218416Z return func(*args, **kwargs) 2025-12-04T09:40:09.0218685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0218765Z self_outputs = self.self( 2025-12-04T09:40:09.0219029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0219101Z return func(*args, **kwargs) 2025-12-04T09:40:09.0219364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:40:09.0219448Z self.value(current_states) 2025-12-04T09:40:09.0219451Z 2025-12-04T09:40:09.0219538Z cudagraph partition due to non gpu ops 2025-12-04T09:40:09.0219648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0219869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0219938Z res = mod(**inputs) 2025-12-04T09:40:09.0220209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0220279Z outputs = self.bert( 2025-12-04T09:40:09.0220547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0220635Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0220898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0220973Z layer_outputs = layer_module( 2025-12-04T09:40:09.0221220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0221304Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0221570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0221642Z return func(*args, **kwargs) 2025-12-04T09:40:09.0221901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0221997Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0222256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0222336Z return func(*args, **kwargs) 2025-12-04T09:40:09.0222596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:40:09.0222671Z self_outputs = self.self( 2025-12-04T09:40:09.0222934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0223009Z return func(*args, **kwargs) 2025-12-04T09:40:09.0223272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:40:09.0223424Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:40:09.0223428Z 2025-12-04T09:40:09.0223579Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0223798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0223867Z res = mod(**inputs) 2025-12-04T09:40:09.0224133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0224213Z outputs = self.bert( 2025-12-04T09:40:09.0224481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0224668Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0224938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0225020Z layer_outputs = layer_module( 2025-12-04T09:40:09.0225262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0225348Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0225614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0225685Z return func(*args, **kwargs) 2025-12-04T09:40:09.0225950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:40:09.0226046Z self_attention_outputs = self.attention( 2025-12-04T09:40:09.0226302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0226382Z return func(*args, **kwargs) 2025-12-04T09:40:09.0226644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:40:09.0226780Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:09.0227051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:40:09.0227141Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0227144Z 2025-12-04T09:40:09.0227260Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0227473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0227541Z res = mod(**inputs) 2025-12-04T09:40:09.0227816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0227890Z outputs = self.bert( 2025-12-04T09:40:09.0228155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0228241Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0228506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0228590Z layer_outputs = layer_module( 2025-12-04T09:40:09.0228829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0228912Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0229175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0229247Z return func(*args, **kwargs) 2025-12-04T09:40:09.0229508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0229610Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0229892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0229982Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0230314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0230446Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0230719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:40:09.0230811Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0230815Z 2025-12-04T09:40:09.0230934Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0231149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0231252Z res = mod(**inputs) 2025-12-04T09:40:09.0231531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0231606Z outputs = self.bert( 2025-12-04T09:40:09.0231875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0231965Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0232229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0232312Z layer_outputs = layer_module( 2025-12-04T09:40:09.0232549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0232634Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0232905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0232980Z return func(*args, **kwargs) 2025-12-04T09:40:09.0233240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0233341Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0233625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0233715Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0234019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:40:09.0234155Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:09.0234435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:40:09.0234565Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:09.0234808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:09.0234897Z return self.act(input) 2025-12-04T09:40:09.0234901Z 2025-12-04T09:40:09.0235013Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0235234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0235302Z res = mod(**inputs) 2025-12-04T09:40:09.0235582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:40:09.0235661Z outputs = self.bert( 2025-12-04T09:40:09.0235928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:40:09.0236013Z encoder_outputs = self.encoder( 2025-12-04T09:40:09.0236279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:40:09.0236355Z layer_outputs = layer_module( 2025-12-04T09:40:09.0236599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:09.0236719Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:09.0236988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:40:09.0237063Z return func(*args, **kwargs) 2025-12-04T09:40:09.0237325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:40:09.0237423Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:09.0237708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:09.0237829Z return forward_fn(*input_tensors) 2025-12-04T09:40:09.0238143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:40:09.0238290Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:09.0238572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:40:09.0238663Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0238667Z 2025-12-04T09:40:09.0238780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0239015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0239086Z res = mod(**inputs) 2025-12-04T09:40:09.0239380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1295, in forward 2025-12-04T09:40:09.0239490Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:40:09.0239775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 752, in forward 2025-12-04T09:40:09.0239910Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:40:09.0240195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 741, in forward 2025-12-04T09:40:09.0240302Z hidden_states = self.transform(hidden_states) 2025-12-04T09:40:09.0240832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 717, in forward 2025-12-04T09:40:09.0240929Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:09.0240933Z 2025-12-04T09:40:09.0241058Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0241286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0241363Z res = mod(**inputs) 2025-12-04T09:40:09.0241645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1295, in forward 2025-12-04T09:40:09.0241749Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:40:09.0242056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 752, in forward 2025-12-04T09:40:09.0242178Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:40:09.0242446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 742, in forward 2025-12-04T09:40:09.0242552Z hidden_states = self.decoder(hidden_states) 2025-12-04T09:40:09.0242556Z 2025-12-04T09:40:09.0242667Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:09.0242880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:09.0242961Z res = mod(**inputs) 2025-12-04T09:40:09.0243236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1300, in forward 2025-12-04T09:40:09.0243459Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:40:09.0243463Z 2025-12-04T09:40:19.6393292Z Compilation time (from dynamo_timed): 17.466924513 2025-12-04T09:40:19.6469095Z pass 2025-12-04T09:40:19.6473032Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:19.6474077Z TIMING: _recursive_pre_grad_passes:0.00652 _recursive_joint_graph_passes:0.70304 _recursive_post_grad_passes:0.07269 async_compile.wait:0.85159 code_gen:9.99065 inductor_compile:11.40482 backend_compile:14.63989 gc:0.00101 entire_frame_compile:17.46692 total_wall_time:17.46692 2025-12-04T09:40:19.6475078Z STATS: call_* op count: 289 | FakeTensorMode.__torch_dispatch__:7149 | FakeTensor.__torch_dispatch__:4342 | ProxyTorchDispatchMode.__torch_dispatch__:1972 2025-12-04T09:40:19.6476935Z Dynamo produced 1 graphs covering 289 ops with 0 graph breaks (0 unique) 2025-12-04T09:40:22.0621627Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:40:22.0622529Z import pynvml # type: ignore[import] 2025-12-04T09:40:25.3827216Z 2025-12-04T09:40:41.5937806Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:40:41.5938347Z loading model: 0it [00:16, ?it/s] 2025-12-04T09:40:41.5975320Z cpu eval BlenderbotForCausalLM 2025-12-04T09:40:41.6152457Z Compilation time (from dynamo_timed): 0 2025-12-04T09:40:41.6152790Z pass_due_to_skip 2025-12-04T09:40:41.6153127Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:41.6153519Z TIMING: total_wall_time:0 2025-12-04T09:40:41.6153716Z STATS: call_* op count: 0 2025-12-04T09:40:41.6153997Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-12-04T09:40:43.8632958Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:40:43.8636166Z import pynvml # type: ignore[import] 2025-12-04T09:40:47.1912119Z 2025-12-04T09:40:54.7160334Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:40:54.7160834Z loading model: 0it [00:07, ?it/s] 2025-12-04T09:40:54.7186879Z cpu eval DebertaV2ForMaskedLM 2025-12-04T09:40:54.7388878Z Compilation time (from dynamo_timed): 0 2025-12-04T09:40:54.7390382Z pass_due_to_skip 2025-12-04T09:40:54.7390855Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:54.7397188Z TIMING: total_wall_time:0 2025-12-04T09:40:54.7397466Z STATS: call_* op count: 0 2025-12-04T09:40:54.7397760Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-12-04T09:40:56.5556690Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:40:56.5557486Z import pynvml # type: ignore[import] 2025-12-04T09:41:00.0679729Z 2025-12-04T09:41:00.7441152Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:41:00.7441463Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:41:00.7450951Z cpu eval DistilBertForMaskedLM 2025-12-04T09:41:00.9083044Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:00.9670076Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:01.0239623Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:05.8930572Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.8930932Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.8931584Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.8931796Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.8932027Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.8932254Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.8932510Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.8932930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.8933306Z res = mod(**inputs) 2025-12-04T09:41:05.8933784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.8934364Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.8934829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.8935368Z return self.transformer( 2025-12-04T09:41:05.8935821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.8936332Z layer_outputs = layer_module( 2025-12-04T09:41:05.8936731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.8937153Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.8937626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.8938097Z sa_output = self.attention( 2025-12-04T09:41:05.8938554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:41:05.8939086Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:41:05.8939292Z 2025-12-04T09:41:05.8939412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.8939817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.8940176Z res = mod(**inputs) 2025-12-04T09:41:05.8940610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.8941079Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.8941541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.8942011Z return self.transformer( 2025-12-04T09:41:05.8942462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.8942942Z layer_outputs = layer_module( 2025-12-04T09:41:05.8943358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.8943773Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.8944234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.8944685Z sa_output = self.attention( 2025-12-04T09:41:05.8945126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:41:05.8945647Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.8945852Z 2025-12-04T09:41:05.8945970Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.8946370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.8946725Z res = mod(**inputs) 2025-12-04T09:41:05.8947506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.8948134Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.8948595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.8949054Z return self.transformer( 2025-12-04T09:41:05.8949485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.8949934Z layer_outputs = layer_module( 2025-12-04T09:41:05.8950313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.8950769Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.8951227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.8951679Z sa_output = self.attention( 2025-12-04T09:41:05.8952187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:41:05.8952673Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.8952870Z 2025-12-04T09:41:05.8952958Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.8953218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.8953603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.8953953Z res = mod(**inputs) 2025-12-04T09:41:05.8954348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.8954769Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.8955175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.8955594Z return self.transformer( 2025-12-04T09:41:05.8956000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.8956418Z layer_outputs = layer_module( 2025-12-04T09:41:05.8956813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.8957210Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.8957671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.8958118Z sa_output = self.attention( 2025-12-04T09:41:05.8958568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:41:05.8959100Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:05.8959304Z 2025-12-04T09:41:05.8959430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.8959824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.8960182Z res = mod(**inputs) 2025-12-04T09:41:05.8960679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.8961146Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.8961609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.8962070Z return self.transformer( 2025-12-04T09:41:05.8962525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.8962968Z layer_outputs = layer_module( 2025-12-04T09:41:05.8963409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.8963819Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.8964238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.8964659Z sa_output = self.attention( 2025-12-04T09:41:05.8965068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:41:05.8965496Z attn_output = self.out_lin(attn_output) 2025-12-04T09:41:05.8965636Z 2025-12-04T09:41:05.8965745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.8966150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.8966477Z res = mod(**inputs) 2025-12-04T09:41:05.8966869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.8967285Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.8967696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.8968110Z return self.transformer( 2025-12-04T09:41:05.8968509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.8968924Z layer_outputs = layer_module( 2025-12-04T09:41:05.8969278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.8969651Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.8970067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.8970524Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.8970981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.8971533Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.8972055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.8972459Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.8972880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:41:05.8973298Z x = self.lin1(input) 2025-12-04T09:41:05.8973412Z 2025-12-04T09:41:05.8973654Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.8974030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.8974348Z res = mod(**inputs) 2025-12-04T09:41:05.8974728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.8975139Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.8975542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.8975949Z return self.transformer( 2025-12-04T09:41:05.8976341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.8976751Z layer_outputs = layer_module( 2025-12-04T09:41:05.8977095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.8977452Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.8977902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.8978349Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.8978792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.8979323Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.8979837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.8980234Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.8981394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:41:05.8981795Z x = self.activation(x) 2025-12-04T09:41:05.8982125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:05.8982468Z return self.act(input) 2025-12-04T09:41:05.8982581Z 2025-12-04T09:41:05.8982690Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.8983040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.8983356Z res = mod(**inputs) 2025-12-04T09:41:05.8983736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.8984144Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.8984556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.8984971Z return self.transformer( 2025-12-04T09:41:05.8985372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.8985806Z layer_outputs = layer_module( 2025-12-04T09:41:05.8986178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.8986566Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.8987013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.8987506Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.8987988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.8988571Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.8989113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.8989534Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.8989978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:41:05.8990407Z x = self.lin2(x) 2025-12-04T09:41:05.8990515Z 2025-12-04T09:41:05.8990626Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.8991012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.8991340Z res = mod(**inputs) 2025-12-04T09:41:05.8991753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.8992209Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.8992654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.8993102Z return self.transformer( 2025-12-04T09:41:05.8993563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.8994084Z layer_outputs = layer_module( 2025-12-04T09:41:05.8994461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.8994850Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.8995297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.8995736Z sa_output = self.attention( 2025-12-04T09:41:05.8996204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:41:05.8996705Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:41:05.8996905Z 2025-12-04T09:41:05.8997018Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.8997409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.8997753Z res = mod(**inputs) 2025-12-04T09:41:05.8998173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.8998633Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.8999086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.8999541Z return self.transformer( 2025-12-04T09:41:05.8999967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9000637Z layer_outputs = layer_module( 2025-12-04T09:41:05.9001027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9001427Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9001875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9002326Z sa_output = self.attention( 2025-12-04T09:41:05.9002767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:41:05.9003230Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.9003415Z 2025-12-04T09:41:05.9003522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9003896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9004224Z res = mod(**inputs) 2025-12-04T09:41:05.9004606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9005029Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9005445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9005854Z return self.transformer( 2025-12-04T09:41:05.9006257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9006674Z layer_outputs = layer_module( 2025-12-04T09:41:05.9007026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9007396Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9007817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9008234Z sa_output = self.attention( 2025-12-04T09:41:05.9008698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:41:05.9009246Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.9009443Z 2025-12-04T09:41:05.9009531Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.9009794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9010175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9010524Z res = mod(**inputs) 2025-12-04T09:41:05.9010944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9011416Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9011823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9012248Z return self.transformer( 2025-12-04T09:41:05.9012679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9013111Z layer_outputs = layer_module( 2025-12-04T09:41:05.9013486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9013878Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9014321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9014757Z sa_output = self.attention( 2025-12-04T09:41:05.9015163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:41:05.9015643Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:05.9015830Z 2025-12-04T09:41:05.9015942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9016303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9016633Z res = mod(**inputs) 2025-12-04T09:41:05.9017022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9017435Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9017860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9018306Z return self.transformer( 2025-12-04T09:41:05.9018710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9019144Z layer_outputs = layer_module( 2025-12-04T09:41:05.9019515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9019915Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9020350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9020787Z sa_output = self.attention( 2025-12-04T09:41:05.9021222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:41:05.9021681Z attn_output = self.out_lin(attn_output) 2025-12-04T09:41:05.9021828Z 2025-12-04T09:41:05.9021940Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9022331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9022676Z res = mod(**inputs) 2025-12-04T09:41:05.9023093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9023577Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9024016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9024472Z return self.transformer( 2025-12-04T09:41:05.9024900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9025350Z layer_outputs = layer_module( 2025-12-04T09:41:05.9025729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9026151Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9026586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9027066Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9027549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9028131Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9028680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9029109Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9029555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:41:05.9030000Z x = self.lin1(input) 2025-12-04T09:41:05.9030117Z 2025-12-04T09:41:05.9030228Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9030613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9030955Z res = mod(**inputs) 2025-12-04T09:41:05.9031364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9031814Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9032257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9032711Z return self.transformer( 2025-12-04T09:41:05.9033142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9033601Z layer_outputs = layer_module( 2025-12-04T09:41:05.9033981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9034364Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9034814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9035327Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9035810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9036388Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9036941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9037369Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9037830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:41:05.9038282Z x = self.activation(x) 2025-12-04T09:41:05.9038648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:05.9039063Z return self.act(input) 2025-12-04T09:41:05.9039200Z 2025-12-04T09:41:05.9039318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9039702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9040061Z res = mod(**inputs) 2025-12-04T09:41:05.9040572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9041043Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9041501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9041997Z return self.transformer( 2025-12-04T09:41:05.9042438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9042894Z layer_outputs = layer_module( 2025-12-04T09:41:05.9043280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9043747Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9044207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9044716Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9045215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9045807Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9046365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9046806Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9047426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:41:05.9047884Z x = self.lin2(x) 2025-12-04T09:41:05.9047998Z 2025-12-04T09:41:05.9048113Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9048513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9048871Z res = mod(**inputs) 2025-12-04T09:41:05.9049260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9049685Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9050101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9050520Z return self.transformer( 2025-12-04T09:41:05.9050919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9051339Z layer_outputs = layer_module( 2025-12-04T09:41:05.9051699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9052073Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9052490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9052910Z sa_output = self.attention( 2025-12-04T09:41:05.9053315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:41:05.9053779Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:41:05.9053970Z 2025-12-04T09:41:05.9054076Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9054534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9054863Z res = mod(**inputs) 2025-12-04T09:41:05.9055253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9055686Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9056118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9056550Z return self.transformer( 2025-12-04T09:41:05.9056956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9057465Z layer_outputs = layer_module( 2025-12-04T09:41:05.9057822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9058190Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9058617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9059034Z sa_output = self.attention( 2025-12-04T09:41:05.9059441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:41:05.9059905Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.9060092Z 2025-12-04T09:41:05.9060199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9060570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9060900Z res = mod(**inputs) 2025-12-04T09:41:05.9061289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9061709Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9062147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9062581Z return self.transformer( 2025-12-04T09:41:05.9063014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9063456Z layer_outputs = layer_module( 2025-12-04T09:41:05.9063841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9064211Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9064635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9065052Z sa_output = self.attention( 2025-12-04T09:41:05.9065455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:41:05.9065928Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.9066116Z 2025-12-04T09:41:05.9066197Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.9066441Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9066798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9067129Z res = mod(**inputs) 2025-12-04T09:41:05.9067527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9067941Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9068341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9068750Z return self.transformer( 2025-12-04T09:41:05.9069225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9069639Z layer_outputs = layer_module( 2025-12-04T09:41:05.9069993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9070370Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9070832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9071279Z sa_output = self.attention( 2025-12-04T09:41:05.9071732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:41:05.9072218Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:05.9072416Z 2025-12-04T09:41:05.9072526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9072883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9073201Z res = mod(**inputs) 2025-12-04T09:41:05.9073584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9073988Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9074398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9074816Z return self.transformer( 2025-12-04T09:41:05.9075224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9075637Z layer_outputs = layer_module( 2025-12-04T09:41:05.9075995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9076370Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9076802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9077253Z sa_output = self.attention( 2025-12-04T09:41:05.9077687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:41:05.9078150Z attn_output = self.out_lin(attn_output) 2025-12-04T09:41:05.9078300Z 2025-12-04T09:41:05.9078412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9078805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9079149Z res = mod(**inputs) 2025-12-04T09:41:05.9079569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9080013Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9080538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9080991Z return self.transformer( 2025-12-04T09:41:05.9081420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9081868Z layer_outputs = layer_module( 2025-12-04T09:41:05.9082222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9082616Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9083062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9083551Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9084080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9084663Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9085216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9085644Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9086094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:41:05.9086568Z x = self.lin1(input) 2025-12-04T09:41:05.9086691Z 2025-12-04T09:41:05.9086805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9087190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9087536Z res = mod(**inputs) 2025-12-04T09:41:05.9087956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9088402Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9088841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9089281Z return self.transformer( 2025-12-04T09:41:05.9089704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9090147Z layer_outputs = layer_module( 2025-12-04T09:41:05.9090528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9090911Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9091355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9091841Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9092319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9092885Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9093438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9093862Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9094284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:41:05.9094697Z x = self.activation(x) 2025-12-04T09:41:05.9095034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:05.9095382Z return self.act(input) 2025-12-04T09:41:05.9095497Z 2025-12-04T09:41:05.9095603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9095971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9096296Z res = mod(**inputs) 2025-12-04T09:41:05.9096687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9097101Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9097513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9097930Z return self.transformer( 2025-12-04T09:41:05.9098351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9098783Z layer_outputs = layer_module( 2025-12-04T09:41:05.9099204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9099603Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9100053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9100516Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9100973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9101517Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9102062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9102465Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9102889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:41:05.9103295Z x = self.lin2(x) 2025-12-04T09:41:05.9103396Z 2025-12-04T09:41:05.9103502Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9103870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9104195Z res = mod(**inputs) 2025-12-04T09:41:05.9104579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9105011Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9105425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9105840Z return self.transformer( 2025-12-04T09:41:05.9106237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9106651Z layer_outputs = layer_module( 2025-12-04T09:41:05.9107003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9107366Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9107791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9108208Z sa_output = self.attention( 2025-12-04T09:41:05.9108611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:41:05.9109079Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:41:05.9109269Z 2025-12-04T09:41:05.9109377Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9109749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9110078Z res = mod(**inputs) 2025-12-04T09:41:05.9110461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9110881Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9111293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9111699Z return self.transformer( 2025-12-04T09:41:05.9112103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9112524Z layer_outputs = layer_module( 2025-12-04T09:41:05.9117024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9117433Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9117961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9118416Z sa_output = self.attention( 2025-12-04T09:41:05.9118848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:41:05.9119361Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.9119554Z 2025-12-04T09:41:05.9119678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9120086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9120550Z res = mod(**inputs) 2025-12-04T09:41:05.9120991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9121490Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9121941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9122382Z return self.transformer( 2025-12-04T09:41:05.9122811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9123254Z layer_outputs = layer_module( 2025-12-04T09:41:05.9123630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9124030Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9124478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9124913Z sa_output = self.attention( 2025-12-04T09:41:05.9125352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:41:05.9125845Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.9126036Z 2025-12-04T09:41:05.9126129Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.9126381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9126760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9127103Z res = mod(**inputs) 2025-12-04T09:41:05.9127517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9127960Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9128407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9128856Z return self.transformer( 2025-12-04T09:41:05.9129286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9129730Z layer_outputs = layer_module( 2025-12-04T09:41:05.9130105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9130487Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9130986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9131437Z sa_output = self.attention( 2025-12-04T09:41:05.9131881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:41:05.9132381Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:05.9132669Z 2025-12-04T09:41:05.9132780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9133215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9133568Z res = mod(**inputs) 2025-12-04T09:41:05.9133990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9134440Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9134880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9135321Z return self.transformer( 2025-12-04T09:41:05.9135760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9136217Z layer_outputs = layer_module( 2025-12-04T09:41:05.9136597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9136981Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9137429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9137868Z sa_output = self.attention( 2025-12-04T09:41:05.9138295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:41:05.9138740Z attn_output = self.out_lin(attn_output) 2025-12-04T09:41:05.9138896Z 2025-12-04T09:41:05.9139008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9139396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9139735Z res = mod(**inputs) 2025-12-04T09:41:05.9140145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9140590Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9141028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9141458Z return self.transformer( 2025-12-04T09:41:05.9141884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9142322Z layer_outputs = layer_module( 2025-12-04T09:41:05.9142684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9143071Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9143515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9144000Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9144468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9145045Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9145603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9146030Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9146466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:41:05.9146903Z x = self.lin1(input) 2025-12-04T09:41:05.9147022Z 2025-12-04T09:41:05.9147293Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9147704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9148094Z res = mod(**inputs) 2025-12-04T09:41:05.9148544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9148996Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9149430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9149876Z return self.transformer( 2025-12-04T09:41:05.9150306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9150743Z layer_outputs = layer_module( 2025-12-04T09:41:05.9151095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9151528Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9151950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9152401Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9152850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9153395Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9153921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9154315Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9154749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:41:05.9155156Z x = self.activation(x) 2025-12-04T09:41:05.9155487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:05.9155819Z return self.act(input) 2025-12-04T09:41:05.9155937Z 2025-12-04T09:41:05.9156045Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9156413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9156741Z res = mod(**inputs) 2025-12-04T09:41:05.9157152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9157592Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9158026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9158434Z return self.transformer( 2025-12-04T09:41:05.9158840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9159266Z layer_outputs = layer_module( 2025-12-04T09:41:05.9159643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9160027Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9160535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9161045Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9161536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9162119Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9162634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9163066Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9163498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:41:05.9163901Z x = self.lin2(x) 2025-12-04T09:41:05.9164009Z 2025-12-04T09:41:05.9164111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9164469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9164786Z res = mod(**inputs) 2025-12-04T09:41:05.9165168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9165577Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9166001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9166412Z return self.transformer( 2025-12-04T09:41:05.9166811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9167211Z layer_outputs = layer_module( 2025-12-04T09:41:05.9167542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9167895Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9168307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9168705Z sa_output = self.attention( 2025-12-04T09:41:05.9169084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:41:05.9169534Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:41:05.9169712Z 2025-12-04T09:41:05.9169827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9170180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9170503Z res = mod(**inputs) 2025-12-04T09:41:05.9170887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9171299Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9171699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9172116Z return self.transformer( 2025-12-04T09:41:05.9172533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9172945Z layer_outputs = layer_module( 2025-12-04T09:41:05.9173286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9173647Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9174066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9174469Z sa_output = self.attention( 2025-12-04T09:41:05.9174868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:41:05.9175325Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.9175501Z 2025-12-04T09:41:05.9175612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9175962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9176287Z res = mod(**inputs) 2025-12-04T09:41:05.9176671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9177107Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9177544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9177965Z return self.transformer( 2025-12-04T09:41:05.9178367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9178782Z layer_outputs = layer_module( 2025-12-04T09:41:05.9179140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9179501Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9179933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9180343Z sa_output = self.attention( 2025-12-04T09:41:05.9180746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:41:05.9181211Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.9181387Z 2025-12-04T09:41:05.9181499Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.9181733Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9182092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9182414Z res = mod(**inputs) 2025-12-04T09:41:05.9182791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9183208Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9183628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9184052Z return self.transformer( 2025-12-04T09:41:05.9184457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9184875Z layer_outputs = layer_module( 2025-12-04T09:41:05.9185235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9185610Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9186029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9186440Z sa_output = self.attention( 2025-12-04T09:41:05.9186842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:41:05.9187308Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:05.9187503Z 2025-12-04T09:41:05.9187608Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9187968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9188285Z res = mod(**inputs) 2025-12-04T09:41:05.9188666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9189080Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9189490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9189896Z return self.transformer( 2025-12-04T09:41:05.9190307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9190730Z layer_outputs = layer_module( 2025-12-04T09:41:05.9191087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9191489Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9191948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9192359Z sa_output = self.attention( 2025-12-04T09:41:05.9192747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:41:05.9193182Z attn_output = self.out_lin(attn_output) 2025-12-04T09:41:05.9193328Z 2025-12-04T09:41:05.9193430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9193783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9194116Z res = mod(**inputs) 2025-12-04T09:41:05.9194499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9194922Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9195362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9195797Z return self.transformer( 2025-12-04T09:41:05.9196225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9196671Z layer_outputs = layer_module( 2025-12-04T09:41:05.9197035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9197425Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9197870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9198354Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9198828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9199413Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9199980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9200507Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9200975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:41:05.9201433Z x = self.lin1(input) 2025-12-04T09:41:05.9201560Z 2025-12-04T09:41:05.9201684Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9202094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9202441Z res = mod(**inputs) 2025-12-04T09:41:05.9202860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9203309Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9203751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9204207Z return self.transformer( 2025-12-04T09:41:05.9204637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9205081Z layer_outputs = layer_module( 2025-12-04T09:41:05.9205451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9205850Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9206299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9206868Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9207352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9207935Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9208490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9208910Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9209349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:41:05.9209783Z x = self.activation(x) 2025-12-04T09:41:05.9210116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:05.9210452Z return self.act(input) 2025-12-04T09:41:05.9210571Z 2025-12-04T09:41:05.9210682Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9211051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9211379Z res = mod(**inputs) 2025-12-04T09:41:05.9211770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9212176Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9212580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9212972Z return self.transformer( 2025-12-04T09:41:05.9213362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9213783Z layer_outputs = layer_module( 2025-12-04T09:41:05.9214139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9214498Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9214917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9230021Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9230638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9231266Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9231868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9232321Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9232779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:41:05.9233230Z x = self.lin2(x) 2025-12-04T09:41:05.9233355Z 2025-12-04T09:41:05.9233477Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9233888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9234261Z res = mod(**inputs) 2025-12-04T09:41:05.9234698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9235198Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9235649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9236227Z return self.transformer( 2025-12-04T09:41:05.9236673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9237222Z layer_outputs = layer_module( 2025-12-04T09:41:05.9237611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9238015Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9238476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9238938Z sa_output = self.attention( 2025-12-04T09:41:05.9239378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:41:05.9239925Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:41:05.9240128Z 2025-12-04T09:41:05.9240246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9240745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9241122Z res = mod(**inputs) 2025-12-04T09:41:05.9241558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9242037Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9242514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9242989Z return self.transformer( 2025-12-04T09:41:05.9243430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9243949Z layer_outputs = layer_module( 2025-12-04T09:41:05.9244347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9244767Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9245235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9245696Z sa_output = self.attention( 2025-12-04T09:41:05.9246143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:41:05.9246658Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.9246861Z 2025-12-04T09:41:05.9246980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9247580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9247949Z res = mod(**inputs) 2025-12-04T09:41:05.9248377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9248855Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9249329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9249791Z return self.transformer( 2025-12-04T09:41:05.9250230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9250695Z layer_outputs = layer_module( 2025-12-04T09:41:05.9251087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9251490Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9251957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9252418Z sa_output = self.attention( 2025-12-04T09:41:05.9252916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:41:05.9253449Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:41:05.9253645Z 2025-12-04T09:41:05.9253733Z cudagraph partition due to non gpu ops 2025-12-04T09:41:05.9253988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9254364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9254689Z res = mod(**inputs) 2025-12-04T09:41:05.9255088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9255517Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9255957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9256383Z return self.transformer( 2025-12-04T09:41:05.9256793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9257212Z layer_outputs = layer_module( 2025-12-04T09:41:05.9257560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9257930Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9258352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9258756Z sa_output = self.attention( 2025-12-04T09:41:05.9259183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:41:05.9259693Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:05.9259883Z 2025-12-04T09:41:05.9259996Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9260359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9260687Z res = mod(**inputs) 2025-12-04T09:41:05.9261096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9261547Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9261955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9262373Z return self.transformer( 2025-12-04T09:41:05.9262791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9263225Z layer_outputs = layer_module( 2025-12-04T09:41:05.9263600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9264000Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9264427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:41:05.9264849Z sa_output = self.attention( 2025-12-04T09:41:05.9265254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:41:05.9265686Z attn_output = self.out_lin(attn_output) 2025-12-04T09:41:05.9265827Z 2025-12-04T09:41:05.9265939Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9266296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9266634Z res = mod(**inputs) 2025-12-04T09:41:05.9267053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9267527Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9267997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9268417Z return self.transformer( 2025-12-04T09:41:05.9268822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9269257Z layer_outputs = layer_module( 2025-12-04T09:41:05.9269630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9270027Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9270477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9270983Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9271477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9272046Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9272598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9273042Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9273500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:41:05.9273950Z x = self.lin1(input) 2025-12-04T09:41:05.9274069Z 2025-12-04T09:41:05.9274185Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9274573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9274922Z res = mod(**inputs) 2025-12-04T09:41:05.9275337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9275775Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9276216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9276665Z return self.transformer( 2025-12-04T09:41:05.9277091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9277531Z layer_outputs = layer_module( 2025-12-04T09:41:05.9277906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9278299Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9278737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9279220Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9279701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9280278Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9280916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9281369Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9281829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:41:05.9282276Z x = self.activation(x) 2025-12-04T09:41:05.9282646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:05.9283049Z return self.act(input) 2025-12-04T09:41:05.9283169Z 2025-12-04T09:41:05.9283331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9283718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9284070Z res = mod(**inputs) 2025-12-04T09:41:05.9284487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:41:05.9284913Z dlbrt_output = self.distilbert( 2025-12-04T09:41:05.9285320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:41:05.9285746Z return self.transformer( 2025-12-04T09:41:05.9286148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:41:05.9286557Z layer_outputs = layer_module( 2025-12-04T09:41:05.9286916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:05.9287281Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:05.9287699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:41:05.9288142Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:41:05.9288598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:41:05.9289148Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:41:05.9289681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:05.9290084Z return forward_fn(*input_tensors) 2025-12-04T09:41:05.9290516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:41:05.9290925Z x = self.lin2(x) 2025-12-04T09:41:05.9291029Z 2025-12-04T09:41:05.9291135Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9291503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9291826Z res = mod(**inputs) 2025-12-04T09:41:05.9292216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 824, in forward 2025-12-04T09:41:05.9292712Z prediction_logits = self.vocab_transform(hidden_states) # (bs, seq_length, dim) 2025-12-04T09:41:05.9292940Z 2025-12-04T09:41:05.9293047Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9293412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9293738Z res = mod(**inputs) 2025-12-04T09:41:05.9294124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 827, in forward 2025-12-04T09:41:05.9294645Z prediction_logits = self.vocab_projector(prediction_logits) # (bs, seq_length, vocab_size) 2025-12-04T09:41:05.9294885Z 2025-12-04T09:41:05.9294998Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:05.9295347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:05.9295667Z res = mod(**inputs) 2025-12-04T09:41:05.9296052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 831, in forward 2025-12-04T09:41:05.9296592Z mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.size(-1)), labels.view(-1)) 2025-12-04T09:41:05.9296872Z 2025-12-04T09:41:15.1001833Z Compilation time (from dynamo_timed): 13.532512047 2025-12-04T09:41:15.1037565Z pass 2025-12-04T09:41:15.1038822Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:15.1039720Z TIMING: _recursive_pre_grad_passes:0.00559 _recursive_joint_graph_passes:0.26626 _recursive_post_grad_passes:0.04733 async_compile.wait:0.77737 code_gen:9.24073 inductor_compile:9.99581 backend_compile:11.81506 gc:0.0008 entire_frame_compile:13.53251 total_wall_time:13.53251 2025-12-04T09:41:15.1041190Z STATS: call_* op count: 153 | FakeTensorMode.__torch_dispatch__:3980 | FakeTensor.__torch_dispatch__:2344 | ProxyTorchDispatchMode.__torch_dispatch__:1053 2025-12-04T09:41:15.1041764Z Dynamo produced 1 graphs covering 153 ops with 0 graph breaks (0 unique) 2025-12-04T09:41:17.3437354Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:41:17.3438259Z import pynvml # type: ignore[import] 2025-12-04T09:41:20.7689059Z 2025-12-04T09:41:21.7807963Z loading model: 0it [00:00, ?it/s]`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`. 2025-12-04T09:41:21.7808667Z WARNING:transformers.modeling_utils:`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`. 2025-12-04T09:41:21.8141866Z 2025-12-04T09:41:21.8149028Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:41:21.8151970Z cpu eval DistillGPT2 2025-12-04T09:41:22.3105767Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:22.5286466Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:22.7319088Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:29.1566853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1567571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1568006Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1568433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:41:29.1568849Z causal_mask = create_causal_mask( 2025-12-04T09:41:29.1569276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:41:29.1569835Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:41:29.1570385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:41:29.1570859Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:41:29.1571329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 655, in find_packed_sequence_indices 2025-12-04T09:41:29.1571953Z first_dummy_value = position_ids[:, :1] - 1 # We just need the diff on this first value to be 1 2025-12-04T09:41:29.1572205Z 2025-12-04T09:41:29.1572298Z cudagraph partition due to non gpu ops 2025-12-04T09:41:29.1572567Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1573060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1573509Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1573957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1574720Z outputs = block( 2025-12-04T09:41:29.1575076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1575597Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1575997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1576382Z return func(*args, **kwargs) 2025-12-04T09:41:29.1576775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1577189Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1577593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1578020Z return func(*args, **kwargs) 2025-12-04T09:41:29.1578406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:41:29.1578927Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:41:29.1579440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1579878Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1580081Z 2025-12-04T09:41:29.1580198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1580675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1581126Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1581595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:41:29.1582033Z causal_mask = create_causal_mask( 2025-12-04T09:41:29.1582463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:41:29.1583006Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:41:29.1583589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:41:29.1584096Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:41:29.1584594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-12-04T09:41:29.1585111Z position_diff = torch.diff(position_ids, prepend=first_dummy_value, dim=-1) 2025-12-04T09:41:29.1585347Z 2025-12-04T09:41:29.1585441Z cudagraph partition due to non gpu ops 2025-12-04T09:41:29.1585711Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1586182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1586615Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1587046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:41:29.1587468Z causal_mask = create_causal_mask( 2025-12-04T09:41:29.1587871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:41:29.1588419Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:41:29.1588981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:41:29.1589472Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:41:29.1589960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T09:41:29.1590451Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T09:41:29.1590614Z 2025-12-04T09:41:29.1590781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1591197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1591587Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1591971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:41:29.1592344Z causal_mask = create_causal_mask( 2025-12-04T09:41:29.1592716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:41:29.1593229Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:41:29.1593749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:41:29.1594193Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:41:29.1594647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T09:41:29.1595088Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T09:41:29.1595252Z 2025-12-04T09:41:29.1595344Z cudagraph partition due to non gpu ops 2025-12-04T09:41:29.1595581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1596002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1596404Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1596802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1597191Z outputs = block( 2025-12-04T09:41:29.1597534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1597916Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1598298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1598689Z return func(*args, **kwargs) 2025-12-04T09:41:29.1599071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1599482Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1599907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1600310Z return func(*args, **kwargs) 2025-12-04T09:41:29.1600897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:41:29.1601333Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:29.1601837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:29.1602328Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:29.1602512Z 2025-12-04T09:41:29.1602625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1603036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1603430Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1603821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1604186Z outputs = block( 2025-12-04T09:41:29.1604537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1604940Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1605321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1605685Z return func(*args, **kwargs) 2025-12-04T09:41:29.1606054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1606451Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1606839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1607217Z return func(*args, **kwargs) 2025-12-04T09:41:29.1607578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:41:29.1607968Z attn_output = self.c_proj(attn_output) 2025-12-04T09:41:29.1608327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1608718Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1608888Z 2025-12-04T09:41:29.1608998Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1609405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1609790Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1610182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1610544Z outputs = block( 2025-12-04T09:41:29.1610853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1611206Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1611575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1611939Z return func(*args, **kwargs) 2025-12-04T09:41:29.1612288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1612691Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1613096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:41:29.1613484Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:41:29.1613835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1614232Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1614403Z 2025-12-04T09:41:29.1614514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1614924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1615327Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1615704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1616060Z outputs = block( 2025-12-04T09:41:29.1616365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1616724Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1617099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1617481Z return func(*args, **kwargs) 2025-12-04T09:41:29.1617854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1618282Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1618735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:41:29.1619119Z hidden_states = self.act(hidden_states) 2025-12-04T09:41:29.1619472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:41:29.1619933Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:41:29.1620167Z 2025-12-04T09:41:29.1620283Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1620697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1621111Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1621512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1621895Z outputs = block( 2025-12-04T09:41:29.1622233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1622606Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1622993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1623372Z return func(*args, **kwargs) 2025-12-04T09:41:29.1623752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1624176Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1624595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:41:29.1624996Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:41:29.1625374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1625785Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1625961Z 2025-12-04T09:41:29.1626074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1626492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1626896Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1627295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1627666Z outputs = block( 2025-12-04T09:41:29.1628000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1628376Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1628768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1629140Z return func(*args, **kwargs) 2025-12-04T09:41:29.1629520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1629927Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1630318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1630809Z return func(*args, **kwargs) 2025-12-04T09:41:29.1631184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:41:29.1631696Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:41:29.1632180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1632618Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1632800Z 2025-12-04T09:41:29.1632889Z cudagraph partition due to non gpu ops 2025-12-04T09:41:29.1633143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1633566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1633972Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1634371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1634752Z outputs = block( 2025-12-04T09:41:29.1635104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1635496Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1635885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1636260Z return func(*args, **kwargs) 2025-12-04T09:41:29.1636640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1637047Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1637444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1637816Z return func(*args, **kwargs) 2025-12-04T09:41:29.1638191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:41:29.1638608Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:29.1639058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:29.1639552Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:29.1639743Z 2025-12-04T09:41:29.1639853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1640278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1640767Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1641164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1641581Z outputs = block( 2025-12-04T09:41:29.1641945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1642361Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1642779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1643166Z return func(*args, **kwargs) 2025-12-04T09:41:29.1643533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1643930Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1644314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1644687Z return func(*args, **kwargs) 2025-12-04T09:41:29.1645055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:41:29.1645449Z attn_output = self.c_proj(attn_output) 2025-12-04T09:41:29.1645820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1646224Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1646460Z 2025-12-04T09:41:29.1646567Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1647021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1647634Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1648021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1648395Z outputs = block( 2025-12-04T09:41:29.1648721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1649087Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1649459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1649888Z return func(*args, **kwargs) 2025-12-04T09:41:29.1650255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1650658Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1651062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:41:29.1651452Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:41:29.1651807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1652197Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1652376Z 2025-12-04T09:41:29.1652481Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1652891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1653283Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1653660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1654031Z outputs = block( 2025-12-04T09:41:29.1654340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1654682Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1655050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1655410Z return func(*args, **kwargs) 2025-12-04T09:41:29.1655768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1656158Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1656558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:41:29.1656944Z hidden_states = self.act(hidden_states) 2025-12-04T09:41:29.1657298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:41:29.1657743Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:41:29.1657986Z 2025-12-04T09:41:29.1658086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1658487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1658864Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1659240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1659600Z outputs = block( 2025-12-04T09:41:29.1659922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1660317Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1660741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1661116Z return func(*args, **kwargs) 2025-12-04T09:41:29.1661482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1661883Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1662286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:41:29.1662684Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:41:29.1663046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1663456Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1663630Z 2025-12-04T09:41:29.1663731Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1664133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1664512Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1664893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1665263Z outputs = block( 2025-12-04T09:41:29.1665584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1665940Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1666319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1666703Z return func(*args, **kwargs) 2025-12-04T09:41:29.1667080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1667487Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1667888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1668257Z return func(*args, **kwargs) 2025-12-04T09:41:29.1668626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:41:29.1669128Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:41:29.1669599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1670003Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1670179Z 2025-12-04T09:41:29.1670263Z cudagraph partition due to non gpu ops 2025-12-04T09:41:29.1670512Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1670948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1671340Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1671740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1672123Z outputs = block( 2025-12-04T09:41:29.1672452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1672858Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1673248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1673636Z return func(*args, **kwargs) 2025-12-04T09:41:29.1674005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1674496Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1674936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1675320Z return func(*args, **kwargs) 2025-12-04T09:41:29.1675689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:41:29.1676105Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:29.1676566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:29.1677059Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:29.1677263Z 2025-12-04T09:41:29.1677370Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1677804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1678221Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1678621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1679011Z outputs = block( 2025-12-04T09:41:29.1679345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1679726Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1680111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1680588Z return func(*args, **kwargs) 2025-12-04T09:41:29.1681000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1681439Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1681865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1682274Z return func(*args, **kwargs) 2025-12-04T09:41:29.1682683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:41:29.1683084Z attn_output = self.c_proj(attn_output) 2025-12-04T09:41:29.1683461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1683877Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1684054Z 2025-12-04T09:41:29.1684170Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1684597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1685009Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1685417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1685802Z outputs = block( 2025-12-04T09:41:29.1686120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1686487Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1686867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1687231Z return func(*args, **kwargs) 2025-12-04T09:41:29.1687599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1688013Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1688428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:41:29.1689697Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:41:29.1690092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1690488Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1690659Z 2025-12-04T09:41:29.1690765Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1691178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1691573Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1691956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1692341Z outputs = block( 2025-12-04T09:41:29.1692659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1693022Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1693398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1693760Z return func(*args, **kwargs) 2025-12-04T09:41:29.1694126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1694533Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1694925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:41:29.1695310Z hidden_states = self.act(hidden_states) 2025-12-04T09:41:29.1695664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:41:29.1696117Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:41:29.1696344Z 2025-12-04T09:41:29.1696449Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1696867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1697264Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1697648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1698009Z outputs = block( 2025-12-04T09:41:29.1698329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1698687Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1699058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1699433Z return func(*args, **kwargs) 2025-12-04T09:41:29.1699796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1700208Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1700602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:41:29.1700995Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:41:29.1701357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1701751Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1701919Z 2025-12-04T09:41:29.1702022Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1702428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1702842Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1703249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1703621Z outputs = block( 2025-12-04T09:41:29.1703944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1704305Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1704678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1705048Z return func(*args, **kwargs) 2025-12-04T09:41:29.1705411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:41:29.1705847Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:41:29.1706004Z 2025-12-04T09:41:29.1706105Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1706522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1706919Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1707298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1707668Z outputs = block( 2025-12-04T09:41:29.1707989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1708353Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1708725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1709097Z return func(*args, **kwargs) 2025-12-04T09:41:29.1709458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1709850Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1710247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1710621Z return func(*args, **kwargs) 2025-12-04T09:41:29.1711005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:41:29.1711504Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:41:29.1711981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1712391Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1712568Z 2025-12-04T09:41:29.1712659Z cudagraph partition due to non gpu ops 2025-12-04T09:41:29.1712899Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1713327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1713736Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1714161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1714569Z outputs = block( 2025-12-04T09:41:29.1714904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1715280Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1715664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1716050Z return func(*args, **kwargs) 2025-12-04T09:41:29.1716430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1716859Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1717287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1717671Z return func(*args, **kwargs) 2025-12-04T09:41:29.1718072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:41:29.1718493Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:29.1718950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:29.1719449Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:29.1719645Z 2025-12-04T09:41:29.1719783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1720230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1720773Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1721209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1721625Z outputs = block( 2025-12-04T09:41:29.1721989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1722393Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1722813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1723233Z return func(*args, **kwargs) 2025-12-04T09:41:29.1723616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1724027Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1724435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1724816Z return func(*args, **kwargs) 2025-12-04T09:41:29.1725198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:41:29.1725600Z attn_output = self.c_proj(attn_output) 2025-12-04T09:41:29.1725962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1726374Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1726555Z 2025-12-04T09:41:29.1726663Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1727090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1727490Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1727890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1728274Z outputs = block( 2025-12-04T09:41:29.1728607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1728974Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1729362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1729745Z return func(*args, **kwargs) 2025-12-04T09:41:29.1730117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1730541Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1730994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:41:29.1731430Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:41:29.1731789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1732226Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1732403Z 2025-12-04T09:41:29.1732516Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1732995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1733392Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1733787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1734166Z outputs = block( 2025-12-04T09:41:29.1734513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1734889Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1735283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1735670Z return func(*args, **kwargs) 2025-12-04T09:41:29.1736046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1736459Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1736874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:41:29.1737257Z hidden_states = self.act(hidden_states) 2025-12-04T09:41:29.1737612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:41:29.1738069Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:41:29.1738299Z 2025-12-04T09:41:29.1738414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1738829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1739228Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1739622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1740010Z outputs = block( 2025-12-04T09:41:29.1740337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1740713Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1741102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1741481Z return func(*args, **kwargs) 2025-12-04T09:41:29.1741861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1742285Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1742704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:41:29.1743100Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:41:29.1743478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1743876Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1744043Z 2025-12-04T09:41:29.1744153Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1744556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1744948Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1745351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1745719Z outputs = block( 2025-12-04T09:41:29.1746074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1746436Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1746809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1747357Z return func(*args, **kwargs) 2025-12-04T09:41:29.1747738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1748150Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1748605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1748984Z return func(*args, **kwargs) 2025-12-04T09:41:29.1749369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:41:29.1749882Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:41:29.1750349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1750762Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1750943Z 2025-12-04T09:41:29.1751031Z cudagraph partition due to non gpu ops 2025-12-04T09:41:29.1751283Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1751705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1752121Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1752526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1752916Z outputs = block( 2025-12-04T09:41:29.1753247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1753620Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1754011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1754391Z return func(*args, **kwargs) 2025-12-04T09:41:29.1754772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1755179Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1755586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1755961Z return func(*args, **kwargs) 2025-12-04T09:41:29.1756344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:41:29.1756768Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:29.1757223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:29.1757748Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:29.1757950Z 2025-12-04T09:41:29.1758062Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1758514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1758954Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1759389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1759876Z outputs = block( 2025-12-04T09:41:29.1760241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1760770Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1761198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1761623Z return func(*args, **kwargs) 2025-12-04T09:41:29.1762007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1762412Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1762811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1763211Z return func(*args, **kwargs) 2025-12-04T09:41:29.1763578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:41:29.1763983Z attn_output = self.c_proj(attn_output) 2025-12-04T09:41:29.1764357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1764767Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1764943Z 2025-12-04T09:41:29.1765052Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1765475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1765882Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1766270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1766646Z outputs = block( 2025-12-04T09:41:29.1766980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1767350Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1767714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1768077Z return func(*args, **kwargs) 2025-12-04T09:41:29.1768434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1768847Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1769249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:41:29.1769709Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:41:29.1770078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1770484Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1770663Z 2025-12-04T09:41:29.1770768Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1771188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1771583Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1771963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1772333Z outputs = block( 2025-12-04T09:41:29.1772656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1773009Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1773389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1773764Z return func(*args, **kwargs) 2025-12-04T09:41:29.1774132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1774557Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1774992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:41:29.1775377Z hidden_states = self.act(hidden_states) 2025-12-04T09:41:29.1775731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:41:29.1776181Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:41:29.1776422Z 2025-12-04T09:41:29.1776532Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1776977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1777368Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1777748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1778116Z outputs = block( 2025-12-04T09:41:29.1778441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1778791Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1779167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1779539Z return func(*args, **kwargs) 2025-12-04T09:41:29.1779912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1780320Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1780725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:41:29.1781125Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:41:29.1781491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1781885Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1782059Z 2025-12-04T09:41:29.1782161Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1782572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1782958Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1783346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1783721Z outputs = block( 2025-12-04T09:41:29.1784048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1784408Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1784798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1785176Z return func(*args, **kwargs) 2025-12-04T09:41:29.1785535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:41:29.1785961Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:41:29.1786128Z 2025-12-04T09:41:29.1786234Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1786650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1787041Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1787431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1787839Z outputs = block( 2025-12-04T09:41:29.1788242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1788612Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1789004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1789398Z return func(*args, **kwargs) 2025-12-04T09:41:29.1789758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1790158Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1790553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1790950Z return func(*args, **kwargs) 2025-12-04T09:41:29.1791315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:41:29.1791818Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:41:29.1792291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1792695Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1792866Z 2025-12-04T09:41:29.1792956Z cudagraph partition due to non gpu ops 2025-12-04T09:41:29.1793181Z cudagraph partition due to non gpu ops 2025-12-04T09:41:29.1793423Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1793835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1794252Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1794642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1795017Z outputs = block( 2025-12-04T09:41:29.1795336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1795696Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1796077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1796453Z return func(*args, **kwargs) 2025-12-04T09:41:29.1796833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1797235Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1797631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1798029Z return func(*args, **kwargs) 2025-12-04T09:41:29.1798437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:41:29.1798885Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:29.1799357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:29.1799875Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:29.1800075Z 2025-12-04T09:41:29.1800185Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1800722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1801166Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1801616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1802049Z outputs = block( 2025-12-04T09:41:29.1802432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1802860Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1803273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1803742Z return func(*args, **kwargs) 2025-12-04T09:41:29.1804140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:41:29.1804584Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:41:29.1805008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1805424Z return func(*args, **kwargs) 2025-12-04T09:41:29.1805815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:41:29.1806254Z attn_output = self.c_proj(attn_output) 2025-12-04T09:41:29.1806644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1807074Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1807258Z 2025-12-04T09:41:29.1807371Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1807814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1808253Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1808662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1809063Z outputs = block( 2025-12-04T09:41:29.1809418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1809779Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1810159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1810531Z return func(*args, **kwargs) 2025-12-04T09:41:29.1810904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1811324Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1811726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:41:29.1812121Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:41:29.1812495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1812885Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1813059Z 2025-12-04T09:41:29.1813165Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1813578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1813974Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1814354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1814727Z outputs = block( 2025-12-04T09:41:29.1815044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1815399Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1815763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1816134Z return func(*args, **kwargs) 2025-12-04T09:41:29.1816505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1816948Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1817389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:41:29.1817772Z hidden_states = self.act(hidden_states) 2025-12-04T09:41:29.1818122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:41:29.1818562Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:41:29.1818795Z 2025-12-04T09:41:29.1818898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1819310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:41:29.1819718Z transformer_outputs = self.transformer( 2025-12-04T09:41:29.1820099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:41:29.1820471Z outputs = block( 2025-12-04T09:41:29.1820800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:29.1821160Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:29.1821545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:29.1821929Z return func(*args, **kwargs) 2025-12-04T09:41:29.1822307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:41:29.1822718Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:41:29.1823133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:41:29.1823548Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:41:29.1823904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:41:29.1824300Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:41:29.1824476Z 2025-12-04T09:41:29.1824582Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:29.1825004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1092, in forward 2025-12-04T09:41:29.1825437Z logits = self.lm_head(hidden_states[:, slice_indices, :]) 2025-12-04T09:41:29.1825611Z 2025-12-04T09:41:38.7294308Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:38.7294869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:41:38.7295400Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:41:38.7295898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:41:38.7296426Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:41:38.7296687Z 2025-12-04T09:41:39.8614399Z Compilation time (from dynamo_timed): 16.363828827 2025-12-04T09:41:39.8764877Z pass 2025-12-04T09:41:39.8766889Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:39.8767797Z TIMING: gc:0.00345 entire_frame_compile:16.36383 _recursive_pre_grad_passes:0.00712 _recursive_joint_graph_passes:0.23651 _recursive_post_grad_passes:0.04866 async_compile.wait:1.48365 code_gen:9.75253 inductor_compile:10.46044 backend_compile:12.41497 total_wall_time:16.36383 2025-12-04T09:41:39.8768894Z STATS: call_* op count: 311 | FakeTensorMode.__torch_dispatch__:4780 | FakeTensor.__torch_dispatch__:2291 | ProxyTorchDispatchMode.__torch_dispatch__:916 2025-12-04T09:41:39.8769667Z Dynamo produced 2 graphs covering 311 ops with 2 graph breaks (1 unique) 2025-12-04T09:41:42.0340986Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:41:42.0341843Z import pynvml # type: ignore[import] 2025-12-04T09:41:45.3391633Z 2025-12-04T09:41:45.3408881Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T09:41:45.3409698Z WARNING:transformers.models.electra.modeling_electra:If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T09:41:45.5470783Z 2025-12-04T09:41:45.5471678Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:41:45.5481809Z cpu eval ElectraForCausalLM 2025-12-04T09:41:45.7030115Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:45.7895069Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:45.8756008Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:53.9779073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9779766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9780349Z res = mod(**inputs) 2025-12-04T09:41:53.9781481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9782098Z outputs = self.electra( 2025-12-04T09:41:53.9782553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 787, in forward 2025-12-04T09:41:53.9783053Z hidden_states = self.embeddings_project(hidden_states) 2025-12-04T09:41:53.9783256Z 2025-12-04T09:41:53.9783402Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9783819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9784194Z res = mod(**inputs) 2025-12-04T09:41:53.9784615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9785049Z outputs = self.electra( 2025-12-04T09:41:53.9785434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9785863Z hidden_states = self.encoder( 2025-12-04T09:41:53.9786307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9786760Z layer_outputs = layer_module( 2025-12-04T09:41:53.9787154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9787556Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9787984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9788390Z return func(*args, **kwargs) 2025-12-04T09:41:53.9788802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9789234Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9789667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9790072Z return func(*args, **kwargs) 2025-12-04T09:41:53.9790480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9791231Z self_outputs = self.self( 2025-12-04T09:41:53.9791765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9792184Z return func(*args, **kwargs) 2025-12-04T09:41:53.9792608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:53.9793052Z query_layer = self.query(hidden_states) 2025-12-04T09:41:53.9793207Z 2025-12-04T09:41:53.9793316Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9793682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9794107Z res = mod(**inputs) 2025-12-04T09:41:53.9794475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9794876Z outputs = self.electra( 2025-12-04T09:41:53.9795256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9795652Z hidden_states = self.encoder( 2025-12-04T09:41:53.9796074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9796513Z layer_outputs = layer_module( 2025-12-04T09:41:53.9796900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9797322Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9797756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9798181Z return func(*args, **kwargs) 2025-12-04T09:41:53.9798614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9799074Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9799510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9799926Z return func(*args, **kwargs) 2025-12-04T09:41:53.9800607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9801077Z self_outputs = self.self( 2025-12-04T09:41:53.9801509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9801936Z return func(*args, **kwargs) 2025-12-04T09:41:53.9802406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:53.9802863Z key_layer = self.key(current_states) 2025-12-04T09:41:53.9803020Z 2025-12-04T09:41:53.9803138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9803539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9803893Z res = mod(**inputs) 2025-12-04T09:41:53.9804304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9804727Z outputs = self.electra( 2025-12-04T09:41:53.9805132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9805534Z hidden_states = self.encoder( 2025-12-04T09:41:53.9805952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9806378Z layer_outputs = layer_module( 2025-12-04T09:41:53.9806746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9807184Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9807631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9808051Z return func(*args, **kwargs) 2025-12-04T09:41:53.9808482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9808928Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9809346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9809763Z return func(*args, **kwargs) 2025-12-04T09:41:53.9810204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9810614Z self_outputs = self.self( 2025-12-04T09:41:53.9811013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9811444Z return func(*args, **kwargs) 2025-12-04T09:41:53.9811863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:53.9812306Z value_layer = self.value(current_states) 2025-12-04T09:41:53.9812452Z 2025-12-04T09:41:53.9812548Z cudagraph partition due to non gpu ops 2025-12-04T09:41:53.9812774Z cudagraph partition due to non gpu ops 2025-12-04T09:41:53.9813026Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9813414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9813759Z res = mod(**inputs) 2025-12-04T09:41:53.9814159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9814585Z outputs = self.electra( 2025-12-04T09:41:53.9814995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9815419Z hidden_states = self.encoder( 2025-12-04T09:41:53.9815858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9816292Z layer_outputs = layer_module( 2025-12-04T09:41:53.9816665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9817051Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9817460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9817860Z return func(*args, **kwargs) 2025-12-04T09:41:53.9818265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9818679Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9819092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9819494Z return func(*args, **kwargs) 2025-12-04T09:41:53.9819897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:53.9820383Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:53.9820867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:53.9821305Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:53.9821456Z 2025-12-04T09:41:53.9821569Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9822008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9822356Z res = mod(**inputs) 2025-12-04T09:41:53.9822788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9823222Z outputs = self.electra( 2025-12-04T09:41:53.9823632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9824055Z hidden_states = self.encoder( 2025-12-04T09:41:53.9824461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9824882Z layer_outputs = layer_module( 2025-12-04T09:41:53.9825277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9825664Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9826072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9826474Z return func(*args, **kwargs) 2025-12-04T09:41:53.9826889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:53.9827313Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:53.9827721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:53.9828127Z return forward_fn(*input_tensors) 2025-12-04T09:41:53.9828561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:53.9829043Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:53.9829492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:53.9829900Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:53.9830039Z 2025-12-04T09:41:53.9830146Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9830507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9830834Z res = mod(**inputs) 2025-12-04T09:41:53.9831210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9831621Z outputs = self.electra( 2025-12-04T09:41:53.9832046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9832466Z hidden_states = self.encoder( 2025-12-04T09:41:53.9832891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9833321Z layer_outputs = layer_module( 2025-12-04T09:41:53.9833704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9834093Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9834498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9835218Z return func(*args, **kwargs) 2025-12-04T09:41:53.9835632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:53.9836074Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:53.9836497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:53.9836922Z return forward_fn(*input_tensors) 2025-12-04T09:41:53.9837414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:53.9837971Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:53.9838463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:53.9838930Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:53.9839344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:53.9839710Z return self.act(input) 2025-12-04T09:41:53.9839840Z 2025-12-04T09:41:53.9839954Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9840460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9840847Z res = mod(**inputs) 2025-12-04T09:41:53.9841264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9841841Z outputs = self.electra( 2025-12-04T09:41:53.9842306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9842729Z hidden_states = self.encoder( 2025-12-04T09:41:53.9843143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9843568Z layer_outputs = layer_module( 2025-12-04T09:41:53.9843951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9844341Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9844757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9845167Z return func(*args, **kwargs) 2025-12-04T09:41:53.9845582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:53.9846022Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:53.9846461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:53.9846891Z return forward_fn(*input_tensors) 2025-12-04T09:41:53.9847706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:53.9848254Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:53.9848763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:53.9849220Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:53.9849379Z 2025-12-04T09:41:53.9849504Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9849922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9850283Z res = mod(**inputs) 2025-12-04T09:41:53.9850698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9851138Z outputs = self.electra( 2025-12-04T09:41:53.9851533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9851948Z hidden_states = self.encoder( 2025-12-04T09:41:53.9852348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9852761Z layer_outputs = layer_module( 2025-12-04T09:41:53.9853131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9853572Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9854023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9854421Z return func(*args, **kwargs) 2025-12-04T09:41:53.9854810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9855224Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9855616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9856002Z return func(*args, **kwargs) 2025-12-04T09:41:53.9856420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9856817Z self_outputs = self.self( 2025-12-04T09:41:53.9857193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9857573Z return func(*args, **kwargs) 2025-12-04T09:41:53.9857960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:53.9858362Z query_layer = self.query(hidden_states) 2025-12-04T09:41:53.9858510Z 2025-12-04T09:41:53.9858618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9858992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9859320Z res = mod(**inputs) 2025-12-04T09:41:53.9859699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9860102Z outputs = self.electra( 2025-12-04T09:41:53.9860483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9860876Z hidden_states = self.encoder( 2025-12-04T09:41:53.9861274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9861678Z layer_outputs = layer_module( 2025-12-04T09:41:53.9862025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9862395Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9862794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9863177Z return func(*args, **kwargs) 2025-12-04T09:41:53.9863559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9863977Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9864373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9864756Z return func(*args, **kwargs) 2025-12-04T09:41:53.9865140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9865539Z self_outputs = self.self( 2025-12-04T09:41:53.9865909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9866278Z return func(*args, **kwargs) 2025-12-04T09:41:53.9866675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:53.9867075Z key_layer = self.key(current_states) 2025-12-04T09:41:53.9867212Z 2025-12-04T09:41:53.9867326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9867715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9868046Z res = mod(**inputs) 2025-12-04T09:41:53.9868461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9868864Z outputs = self.electra( 2025-12-04T09:41:53.9869269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9869668Z hidden_states = self.encoder( 2025-12-04T09:41:53.9870075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9870494Z layer_outputs = layer_module( 2025-12-04T09:41:53.9870911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9871294Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9871674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9872058Z return func(*args, **kwargs) 2025-12-04T09:41:53.9872449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9872871Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9873245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9873624Z return func(*args, **kwargs) 2025-12-04T09:41:53.9874029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9874425Z self_outputs = self.self( 2025-12-04T09:41:53.9874780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9875151Z return func(*args, **kwargs) 2025-12-04T09:41:53.9875544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:53.9875964Z value_layer = self.value(current_states) 2025-12-04T09:41:53.9876118Z 2025-12-04T09:41:53.9876204Z cudagraph partition due to non gpu ops 2025-12-04T09:41:53.9876437Z cudagraph partition due to non gpu ops 2025-12-04T09:41:53.9876695Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9877077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9877427Z res = mod(**inputs) 2025-12-04T09:41:53.9877828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9878248Z outputs = self.electra( 2025-12-04T09:41:53.9878678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9879100Z hidden_states = self.encoder( 2025-12-04T09:41:53.9879534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9879952Z layer_outputs = layer_module( 2025-12-04T09:41:53.9880328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9880813Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9881224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9881633Z return func(*args, **kwargs) 2025-12-04T09:41:53.9882056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9882481Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9882886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9883263Z return func(*args, **kwargs) 2025-12-04T09:41:53.9883654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:53.9884164Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:53.9884618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:53.9885032Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:53.9885181Z 2025-12-04T09:41:53.9885325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9885712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9886065Z res = mod(**inputs) 2025-12-04T09:41:53.9886466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9886868Z outputs = self.electra( 2025-12-04T09:41:53.9887256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9887667Z hidden_states = self.encoder( 2025-12-04T09:41:53.9888071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9888483Z layer_outputs = layer_module( 2025-12-04T09:41:53.9888845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9889228Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9889627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9890010Z return func(*args, **kwargs) 2025-12-04T09:41:53.9890416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:53.9890844Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:53.9891260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:53.9891661Z return forward_fn(*input_tensors) 2025-12-04T09:41:53.9892108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:53.9892603Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:53.9893060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:53.9893477Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:53.9893627Z 2025-12-04T09:41:53.9893736Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9894114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9894444Z res = mod(**inputs) 2025-12-04T09:41:53.9894832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9895240Z outputs = self.electra( 2025-12-04T09:41:53.9895633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9896137Z hidden_states = self.encoder( 2025-12-04T09:41:53.9896537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9896942Z layer_outputs = layer_module( 2025-12-04T09:41:53.9897317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9897722Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9898117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9898503Z return func(*args, **kwargs) 2025-12-04T09:41:53.9898888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:53.9899314Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:53.9899734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:53.9900184Z return forward_fn(*input_tensors) 2025-12-04T09:41:53.9900647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:53.9901168Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:53.9901625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:53.9902061Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:53.9902460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:53.9902815Z return self.act(input) 2025-12-04T09:41:53.9902931Z 2025-12-04T09:41:53.9903047Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9903410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9903742Z res = mod(**inputs) 2025-12-04T09:41:53.9904121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9904522Z outputs = self.electra( 2025-12-04T09:41:53.9904911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9905312Z hidden_states = self.encoder( 2025-12-04T09:41:53.9905710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9906135Z layer_outputs = layer_module( 2025-12-04T09:41:53.9906503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9906871Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9907254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9907622Z return func(*args, **kwargs) 2025-12-04T09:41:53.9908006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:53.9908413Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:53.9908808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:53.9909206Z return forward_fn(*input_tensors) 2025-12-04T09:41:53.9909649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:53.9910131Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:53.9910578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:53.9910984Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:53.9911123Z 2025-12-04T09:41:53.9911237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9911611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9911924Z res = mod(**inputs) 2025-12-04T09:41:53.9912342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9912739Z outputs = self.electra( 2025-12-04T09:41:53.9913110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9913502Z hidden_states = self.encoder( 2025-12-04T09:41:53.9913888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9914277Z layer_outputs = layer_module( 2025-12-04T09:41:53.9914633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9914997Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9915374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9915738Z return func(*args, **kwargs) 2025-12-04T09:41:53.9916112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9916515Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9916897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9917256Z return func(*args, **kwargs) 2025-12-04T09:41:53.9917635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9918028Z self_outputs = self.self( 2025-12-04T09:41:53.9918390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9918759Z return func(*args, **kwargs) 2025-12-04T09:41:53.9919155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:53.9919589Z query_layer = self.query(hidden_states) 2025-12-04T09:41:53.9919739Z 2025-12-04T09:41:53.9919853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9920248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9920682Z res = mod(**inputs) 2025-12-04T09:41:53.9921104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9921539Z outputs = self.electra( 2025-12-04T09:41:53.9921970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9922383Z hidden_states = self.encoder( 2025-12-04T09:41:53.9922758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9923154Z layer_outputs = layer_module( 2025-12-04T09:41:53.9923499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9923863Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9924229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9924623Z return func(*args, **kwargs) 2025-12-04T09:41:53.9925043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9925491Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9925905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9926338Z return func(*args, **kwargs) 2025-12-04T09:41:53.9926799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9927230Z self_outputs = self.self( 2025-12-04T09:41:53.9927635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9928047Z return func(*args, **kwargs) 2025-12-04T09:41:53.9928471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:53.9928912Z key_layer = self.key(current_states) 2025-12-04T09:41:53.9929069Z 2025-12-04T09:41:53.9929206Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9929613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9929969Z res = mod(**inputs) 2025-12-04T09:41:53.9930389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9930830Z outputs = self.electra( 2025-12-04T09:41:53.9931248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9931682Z hidden_states = self.encoder( 2025-12-04T09:41:53.9932114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9932532Z layer_outputs = layer_module( 2025-12-04T09:41:53.9932879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9933238Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9933620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9934002Z return func(*args, **kwargs) 2025-12-04T09:41:53.9934388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9934808Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9935193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9935564Z return func(*args, **kwargs) 2025-12-04T09:41:53.9935940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9936329Z self_outputs = self.self( 2025-12-04T09:41:53.9936697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9937061Z return func(*args, **kwargs) 2025-12-04T09:41:53.9937461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:53.9937877Z value_layer = self.value(current_states) 2025-12-04T09:41:53.9938016Z 2025-12-04T09:41:53.9938110Z cudagraph partition due to non gpu ops 2025-12-04T09:41:53.9938326Z cudagraph partition due to non gpu ops 2025-12-04T09:41:53.9938576Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9938949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9939277Z res = mod(**inputs) 2025-12-04T09:41:53.9939662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9940072Z outputs = self.electra( 2025-12-04T09:41:53.9940462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9940864Z hidden_states = self.encoder( 2025-12-04T09:41:53.9941279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9941671Z layer_outputs = layer_module( 2025-12-04T09:41:53.9942016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9942374Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9942749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9943119Z return func(*args, **kwargs) 2025-12-04T09:41:53.9943494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9943923Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9944311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9944685Z return func(*args, **kwargs) 2025-12-04T09:41:53.9945062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:53.9945510Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:53.9945952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:53.9946353Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:53.9946493Z 2025-12-04T09:41:53.9946596Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9946945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9947465Z res = mod(**inputs) 2025-12-04T09:41:53.9947826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9948215Z outputs = self.electra( 2025-12-04T09:41:53.9948592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9948986Z hidden_states = self.encoder( 2025-12-04T09:41:53.9949369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9949766Z layer_outputs = layer_module( 2025-12-04T09:41:53.9950125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9950503Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9950885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9951261Z return func(*args, **kwargs) 2025-12-04T09:41:53.9951646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:53.9952047Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:53.9952449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:53.9952844Z return forward_fn(*input_tensors) 2025-12-04T09:41:53.9953268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:53.9953740Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:53.9954179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:53.9954587Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:53.9954723Z 2025-12-04T09:41:53.9954880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9955245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9955622Z res = mod(**inputs) 2025-12-04T09:41:53.9955996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9956389Z outputs = self.electra( 2025-12-04T09:41:53.9956779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9957193Z hidden_states = self.encoder( 2025-12-04T09:41:53.9957587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9958013Z layer_outputs = layer_module( 2025-12-04T09:41:53.9958367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9958760Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9959163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9959567Z return func(*args, **kwargs) 2025-12-04T09:41:53.9960000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:53.9960506Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:53.9960954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:53.9961409Z return forward_fn(*input_tensors) 2025-12-04T09:41:53.9961868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:53.9962373Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:53.9962879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:53.9963373Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:53.9963821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:53.9964212Z return self.act(input) 2025-12-04T09:41:53.9964351Z 2025-12-04T09:41:53.9964471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9964885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9965257Z res = mod(**inputs) 2025-12-04T09:41:53.9965674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9966124Z outputs = self.electra( 2025-12-04T09:41:53.9966559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9967015Z hidden_states = self.encoder( 2025-12-04T09:41:53.9967472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9967924Z layer_outputs = layer_module( 2025-12-04T09:41:53.9968321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9968732Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9969148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9969528Z return func(*args, **kwargs) 2025-12-04T09:41:53.9969915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:53.9970362Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:53.9970808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:53.9971239Z return forward_fn(*input_tensors) 2025-12-04T09:41:53.9971668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:53.9972165Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:53.9972625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:53.9973039Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:53.9973181Z 2025-12-04T09:41:53.9973289Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9973682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9974004Z res = mod(**inputs) 2025-12-04T09:41:53.9974366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9974759Z outputs = self.electra( 2025-12-04T09:41:53.9975134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9975529Z hidden_states = self.encoder( 2025-12-04T09:41:53.9975908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9976302Z layer_outputs = layer_module( 2025-12-04T09:41:53.9976651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9977017Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9977389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9977760Z return func(*args, **kwargs) 2025-12-04T09:41:53.9978143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9978542Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9978925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9979294Z return func(*args, **kwargs) 2025-12-04T09:41:53.9979675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9980061Z self_outputs = self.self( 2025-12-04T09:41:53.9980417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9980788Z return func(*args, **kwargs) 2025-12-04T09:41:53.9981158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:53.9981558Z query_layer = self.query(hidden_states) 2025-12-04T09:41:53.9981701Z 2025-12-04T09:41:53.9981807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9982160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9982470Z res = mod(**inputs) 2025-12-04T09:41:53.9982838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9983229Z outputs = self.electra( 2025-12-04T09:41:53.9983595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9983981Z hidden_states = self.encoder( 2025-12-04T09:41:53.9984360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9984765Z layer_outputs = layer_module( 2025-12-04T09:41:53.9985133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9985497Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9985876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9986243Z return func(*args, **kwargs) 2025-12-04T09:41:53.9986613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9987021Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9987411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9987769Z return func(*args, **kwargs) 2025-12-04T09:41:53.9988149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9988539Z self_outputs = self.self( 2025-12-04T09:41:53.9988896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9989254Z return func(*args, **kwargs) 2025-12-04T09:41:53.9989631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:53.9990029Z key_layer = self.key(current_states) 2025-12-04T09:41:53.9990164Z 2025-12-04T09:41:53.9990280Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:53.9990642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:53.9990971Z res = mod(**inputs) 2025-12-04T09:41:53.9991355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:53.9991750Z outputs = self.electra( 2025-12-04T09:41:53.9992139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:53.9992591Z hidden_states = self.encoder( 2025-12-04T09:41:53.9993017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:53.9993427Z layer_outputs = layer_module( 2025-12-04T09:41:53.9993783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:53.9994153Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:53.9994550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9994930Z return func(*args, **kwargs) 2025-12-04T09:41:53.9995324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:53.9995747Z self_attention_outputs = self.attention( 2025-12-04T09:41:53.9996119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9996489Z return func(*args, **kwargs) 2025-12-04T09:41:53.9996868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:53.9997252Z self_outputs = self.self( 2025-12-04T09:41:53.9997619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:53.9998008Z return func(*args, **kwargs) 2025-12-04T09:41:53.9998431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:53.9998880Z value_layer = self.value(current_states) 2025-12-04T09:41:53.9999033Z 2025-12-04T09:41:53.9999122Z cudagraph partition due to non gpu ops 2025-12-04T09:41:53.9999387Z cudagraph partition due to non gpu ops 2025-12-04T09:41:53.9999643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0000035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0000454Z res = mod(**inputs) 2025-12-04T09:41:54.0000892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0001342Z outputs = self.electra( 2025-12-04T09:41:54.0001786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0002234Z hidden_states = self.encoder( 2025-12-04T09:41:54.0002625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0003021Z layer_outputs = layer_module( 2025-12-04T09:41:54.0003384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0003756Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0004140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0004533Z return func(*args, **kwargs) 2025-12-04T09:41:54.0004912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0005312Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0005688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0006062Z return func(*args, **kwargs) 2025-12-04T09:41:54.0006446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:54.0006910Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:54.0007353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:54.0007755Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0007895Z 2025-12-04T09:41:54.0008006Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0008372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0008702Z res = mod(**inputs) 2025-12-04T09:41:54.0009085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0009487Z outputs = self.electra( 2025-12-04T09:41:54.0009869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0010272Z hidden_states = self.encoder( 2025-12-04T09:41:54.0010669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0011056Z layer_outputs = layer_module( 2025-12-04T09:41:54.0011414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0011796Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0012173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0012535Z return func(*args, **kwargs) 2025-12-04T09:41:54.0012917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0013342Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0013763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0014155Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0014576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0015045Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0015477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:54.0015879Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0016055Z 2025-12-04T09:41:54.0016160Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0016522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0016838Z res = mod(**inputs) 2025-12-04T09:41:54.0017213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0017603Z outputs = self.electra( 2025-12-04T09:41:54.0017973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0018364Z hidden_states = self.encoder( 2025-12-04T09:41:54.0018751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0019140Z layer_outputs = layer_module( 2025-12-04T09:41:54.0019477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0019842Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0020219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0020594Z return func(*args, **kwargs) 2025-12-04T09:41:54.0020970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0021371Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0021769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0022154Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0022572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0023041Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0023474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:54.0023900Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:54.0024285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:54.0024638Z return self.act(input) 2025-12-04T09:41:54.0024749Z 2025-12-04T09:41:54.0024860Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0025216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0025534Z res = mod(**inputs) 2025-12-04T09:41:54.0025907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0026290Z outputs = self.electra( 2025-12-04T09:41:54.0026667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0027085Z hidden_states = self.encoder( 2025-12-04T09:41:54.0027472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0027887Z layer_outputs = layer_module( 2025-12-04T09:41:54.0028237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0028600Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0028973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0029347Z return func(*args, **kwargs) 2025-12-04T09:41:54.0029732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0030165Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0030557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0030950Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0031370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:54.0031851Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:54.0032296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:54.0032713Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0032851Z 2025-12-04T09:41:54.0032967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0033327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0033664Z res = mod(**inputs) 2025-12-04T09:41:54.0034036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0034426Z outputs = self.electra( 2025-12-04T09:41:54.0034795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0035183Z hidden_states = self.encoder( 2025-12-04T09:41:54.0035578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0035975Z layer_outputs = layer_module( 2025-12-04T09:41:54.0036323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0036716Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0037125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0037520Z return func(*args, **kwargs) 2025-12-04T09:41:54.0037953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0038401Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0038818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0039210Z return func(*args, **kwargs) 2025-12-04T09:41:54.0039622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0040070Z self_outputs = self.self( 2025-12-04T09:41:54.0040554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0040975Z return func(*args, **kwargs) 2025-12-04T09:41:54.0041403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:54.0041920Z query_layer = self.query(hidden_states) 2025-12-04T09:41:54.0042065Z 2025-12-04T09:41:54.0042173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0042590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0042918Z res = mod(**inputs) 2025-12-04T09:41:54.0043296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0043689Z outputs = self.electra( 2025-12-04T09:41:54.0044079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0044487Z hidden_states = self.encoder( 2025-12-04T09:41:54.0044883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0045278Z layer_outputs = layer_module( 2025-12-04T09:41:54.0045637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0046022Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0046392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0046761Z return func(*args, **kwargs) 2025-12-04T09:41:54.0047303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0047711Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0048103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0048479Z return func(*args, **kwargs) 2025-12-04T09:41:54.0048870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0049260Z self_outputs = self.self( 2025-12-04T09:41:54.0049634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0050012Z return func(*args, **kwargs) 2025-12-04T09:41:54.0050401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:54.0050802Z key_layer = self.key(current_states) 2025-12-04T09:41:54.0050950Z 2025-12-04T09:41:54.0051057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0051420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0051735Z res = mod(**inputs) 2025-12-04T09:41:54.0052116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0052527Z outputs = self.electra( 2025-12-04T09:41:54.0052929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0053314Z hidden_states = self.encoder( 2025-12-04T09:41:54.0053703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0054094Z layer_outputs = layer_module( 2025-12-04T09:41:54.0054437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0054799Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0055180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0055557Z return func(*args, **kwargs) 2025-12-04T09:41:54.0055933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0056405Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0056843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0057219Z return func(*args, **kwargs) 2025-12-04T09:41:54.0057595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0057986Z self_outputs = self.self( 2025-12-04T09:41:54.0058352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0058719Z return func(*args, **kwargs) 2025-12-04T09:41:54.0059117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:54.0059563Z value_layer = self.value(current_states) 2025-12-04T09:41:54.0059703Z 2025-12-04T09:41:54.0059795Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0060013Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0060261Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0060640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0060950Z res = mod(**inputs) 2025-12-04T09:41:54.0061325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0061717Z outputs = self.electra( 2025-12-04T09:41:54.0062091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0062477Z hidden_states = self.encoder( 2025-12-04T09:41:54.0062864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0063255Z layer_outputs = layer_module( 2025-12-04T09:41:54.0063597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0063959Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0064342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0064716Z return func(*args, **kwargs) 2025-12-04T09:41:54.0065093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0065497Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0065878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0066243Z return func(*args, **kwargs) 2025-12-04T09:41:54.0066626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:54.0067081Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:54.0067528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:54.0067926Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0068071Z 2025-12-04T09:41:54.0068173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0068524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0068844Z res = mod(**inputs) 2025-12-04T09:41:54.0069201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0069584Z outputs = self.electra( 2025-12-04T09:41:54.0069948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0070356Z hidden_states = self.encoder( 2025-12-04T09:41:54.0070796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0071175Z layer_outputs = layer_module( 2025-12-04T09:41:54.0071511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0071854Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0072223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0072584Z return func(*args, **kwargs) 2025-12-04T09:41:54.0072967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0073391Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0073815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0074203Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0074624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0075104Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0075530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:54.0075928Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0076065Z 2025-12-04T09:41:54.0076168Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0076529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0076849Z res = mod(**inputs) 2025-12-04T09:41:54.0077211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0077600Z outputs = self.electra( 2025-12-04T09:41:54.0077975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0078364Z hidden_states = self.encoder( 2025-12-04T09:41:54.0078740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0079135Z layer_outputs = layer_module( 2025-12-04T09:41:54.0079497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0079889Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0080292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0080781Z return func(*args, **kwargs) 2025-12-04T09:41:54.0081220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0081683Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0082136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0082580Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0083035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0083521Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0084016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:54.0084510Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:54.0084972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:54.0085352Z return self.act(input) 2025-12-04T09:41:54.0085517Z 2025-12-04T09:41:54.0085637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0086040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0086392Z res = mod(**inputs) 2025-12-04T09:41:54.0086807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0087246Z outputs = self.electra( 2025-12-04T09:41:54.0087674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0088123Z hidden_states = self.encoder( 2025-12-04T09:41:54.0088551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0088988Z layer_outputs = layer_module( 2025-12-04T09:41:54.0089375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0089786Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0090210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0090628Z return func(*args, **kwargs) 2025-12-04T09:41:54.0091050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0091509Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0091910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0092304Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0092722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:54.0093204Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:54.0093645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:54.0094035Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0094179Z 2025-12-04T09:41:54.0094285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0094641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0094963Z res = mod(**inputs) 2025-12-04T09:41:54.0095325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0095718Z outputs = self.electra( 2025-12-04T09:41:54.0096093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0096487Z hidden_states = self.encoder( 2025-12-04T09:41:54.0096856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0097229Z layer_outputs = layer_module( 2025-12-04T09:41:54.0097567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0097912Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0098282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0098645Z return func(*args, **kwargs) 2025-12-04T09:41:54.0099018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0099439Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0099896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0100282Z return func(*args, **kwargs) 2025-12-04T09:41:54.0100669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0101070Z self_outputs = self.self( 2025-12-04T09:41:54.0101445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0101838Z return func(*args, **kwargs) 2025-12-04T09:41:54.0102215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:54.0102637Z query_layer = self.query(hidden_states) 2025-12-04T09:41:54.0102774Z 2025-12-04T09:41:54.0102900Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0103299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0103643Z res = mod(**inputs) 2025-12-04T09:41:54.0104046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0104474Z outputs = self.electra( 2025-12-04T09:41:54.0104878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0105278Z hidden_states = self.encoder( 2025-12-04T09:41:54.0105677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0106068Z layer_outputs = layer_module( 2025-12-04T09:41:54.0106412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0106782Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0107173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0107548Z return func(*args, **kwargs) 2025-12-04T09:41:54.0107940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0108355Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0108746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0109130Z return func(*args, **kwargs) 2025-12-04T09:41:54.0109511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0109900Z self_outputs = self.self( 2025-12-04T09:41:54.0110264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0110638Z return func(*args, **kwargs) 2025-12-04T09:41:54.0111038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:54.0111440Z key_layer = self.key(current_states) 2025-12-04T09:41:54.0111574Z 2025-12-04T09:41:54.0111681Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0112049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0112377Z res = mod(**inputs) 2025-12-04T09:41:54.0112762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0113160Z outputs = self.electra( 2025-12-04T09:41:54.0113547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0113970Z hidden_states = self.encoder( 2025-12-04T09:41:54.0114429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0114828Z layer_outputs = layer_module( 2025-12-04T09:41:54.0115184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0115554Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0115934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0116313Z return func(*args, **kwargs) 2025-12-04T09:41:54.0116704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0117144Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0117531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0117923Z return func(*args, **kwargs) 2025-12-04T09:41:54.0118349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0118780Z self_outputs = self.self( 2025-12-04T09:41:54.0119175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0119576Z return func(*args, **kwargs) 2025-12-04T09:41:54.0120007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:54.0120534Z value_layer = self.value(current_states) 2025-12-04T09:41:54.0120698Z 2025-12-04T09:41:54.0120787Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0121028Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0121295Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0121695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0122062Z res = mod(**inputs) 2025-12-04T09:41:54.0122453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0122857Z outputs = self.electra( 2025-12-04T09:41:54.0123281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0123790Z hidden_states = self.encoder( 2025-12-04T09:41:54.0124187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0124596Z layer_outputs = layer_module( 2025-12-04T09:41:54.0124959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0125338Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0125726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0126110Z return func(*args, **kwargs) 2025-12-04T09:41:54.0126505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0126921Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0127310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0127695Z return func(*args, **kwargs) 2025-12-04T09:41:54.0128090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:54.0128548Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:54.0129031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:54.0129487Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0129631Z 2025-12-04T09:41:54.0129745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0130106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0130434Z res = mod(**inputs) 2025-12-04T09:41:54.0130814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0131208Z outputs = self.electra( 2025-12-04T09:41:54.0131592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0132011Z hidden_states = self.encoder( 2025-12-04T09:41:54.0132408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0132801Z layer_outputs = layer_module( 2025-12-04T09:41:54.0133157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0133531Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0133918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0134290Z return func(*args, **kwargs) 2025-12-04T09:41:54.0134683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0135097Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0135499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0135901Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0136336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0136819Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0137263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:54.0137689Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0137823Z 2025-12-04T09:41:54.0137936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0138298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0138629Z res = mod(**inputs) 2025-12-04T09:41:54.0139014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0139422Z outputs = self.electra( 2025-12-04T09:41:54.0139803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0140208Z hidden_states = self.encoder( 2025-12-04T09:41:54.0140607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0141006Z layer_outputs = layer_module( 2025-12-04T09:41:54.0141356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0141730Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0142119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0142496Z return func(*args, **kwargs) 2025-12-04T09:41:54.0142889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0143329Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0143773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0144174Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0144593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0145067Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0145512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:54.0145949Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:54.0146367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:54.0146723Z return self.act(input) 2025-12-04T09:41:54.0146840Z 2025-12-04T09:41:54.0146950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0147484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0147820Z res = mod(**inputs) 2025-12-04T09:41:54.0148194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0148590Z outputs = self.electra( 2025-12-04T09:41:54.0148981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0149376Z hidden_states = self.encoder( 2025-12-04T09:41:54.0149759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0150154Z layer_outputs = layer_module( 2025-12-04T09:41:54.0150510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0150886Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0151269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0151649Z return func(*args, **kwargs) 2025-12-04T09:41:54.0152050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0152449Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0152842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0153232Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0153652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:54.0154122Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:54.0154571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:54.0154972Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0155108Z 2025-12-04T09:41:54.0155219Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0155565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0155886Z res = mod(**inputs) 2025-12-04T09:41:54.0156253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0156648Z outputs = self.electra( 2025-12-04T09:41:54.0157014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0157456Z hidden_states = self.encoder( 2025-12-04T09:41:54.0157890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0158274Z layer_outputs = layer_module( 2025-12-04T09:41:54.0158629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0159005Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0159396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0159771Z return func(*args, **kwargs) 2025-12-04T09:41:54.0160167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0160699Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0161119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0161533Z return func(*args, **kwargs) 2025-12-04T09:41:54.0161956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0162366Z self_outputs = self.self( 2025-12-04T09:41:54.0162736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0163121Z return func(*args, **kwargs) 2025-12-04T09:41:54.0163517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:54.0163939Z query_layer = self.query(hidden_states) 2025-12-04T09:41:54.0164083Z 2025-12-04T09:41:54.0164194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0164572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0164909Z res = mod(**inputs) 2025-12-04T09:41:54.0165291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0165699Z outputs = self.electra( 2025-12-04T09:41:54.0166087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0166491Z hidden_states = self.encoder( 2025-12-04T09:41:54.0166883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0167285Z layer_outputs = layer_module( 2025-12-04T09:41:54.0167648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0168021Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0168414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0168793Z return func(*args, **kwargs) 2025-12-04T09:41:54.0169193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0169602Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0170003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0170386Z return func(*args, **kwargs) 2025-12-04T09:41:54.0170789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0171165Z self_outputs = self.self( 2025-12-04T09:41:54.0171522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0171906Z return func(*args, **kwargs) 2025-12-04T09:41:54.0172265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:54.0172679Z key_layer = self.key(current_states) 2025-12-04T09:41:54.0172819Z 2025-12-04T09:41:54.0172924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0173273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0173577Z res = mod(**inputs) 2025-12-04T09:41:54.0173942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0174320Z outputs = self.electra( 2025-12-04T09:41:54.0174674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0175072Z hidden_states = self.encoder( 2025-12-04T09:41:54.0175458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0175839Z layer_outputs = layer_module( 2025-12-04T09:41:54.0176171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0176521Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0176891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0177245Z return func(*args, **kwargs) 2025-12-04T09:41:54.0177607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0177998Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0178367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0178719Z return func(*args, **kwargs) 2025-12-04T09:41:54.0179094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0179478Z self_outputs = self.self( 2025-12-04T09:41:54.0179843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0180190Z return func(*args, **kwargs) 2025-12-04T09:41:54.0180558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:54.0180945Z value_layer = self.value(current_states) 2025-12-04T09:41:54.0181078Z 2025-12-04T09:41:54.0181158Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0181376Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0181611Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0181964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0182277Z res = mod(**inputs) 2025-12-04T09:41:54.0182650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0183040Z outputs = self.electra( 2025-12-04T09:41:54.0183297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0183376Z hidden_states = self.encoder( 2025-12-04T09:41:54.0183632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0183702Z layer_outputs = layer_module( 2025-12-04T09:41:54.0183929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0184011Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0184285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0184396Z return func(*args, **kwargs) 2025-12-04T09:41:54.0184659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0184749Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0184998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0185067Z return func(*args, **kwargs) 2025-12-04T09:41:54.0185336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:54.0185485Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:54.0185747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:54.0185838Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0185842Z 2025-12-04T09:41:54.0185949Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0186154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0186230Z res = mod(**inputs) 2025-12-04T09:41:54.0186487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0186561Z outputs = self.electra( 2025-12-04T09:41:54.0186813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0186889Z hidden_states = self.encoder( 2025-12-04T09:41:54.0187145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0187215Z layer_outputs = layer_module( 2025-12-04T09:41:54.0187438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0187517Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0187751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0187825Z return func(*args, **kwargs) 2025-12-04T09:41:54.0188075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0188165Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0188414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0188491Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0188785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0188904Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0189166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:54.0189247Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0189251Z 2025-12-04T09:41:54.0189352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0189554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0189617Z res = mod(**inputs) 2025-12-04T09:41:54.0189873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0189950Z outputs = self.electra( 2025-12-04T09:41:54.0190201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0190297Z hidden_states = self.encoder( 2025-12-04T09:41:54.0190581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0190651Z layer_outputs = layer_module( 2025-12-04T09:41:54.0190873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0190949Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0191191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0191258Z return func(*args, **kwargs) 2025-12-04T09:41:54.0191534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0191622Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0191874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0191950Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0192237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0192353Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0192611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:54.0192719Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:54.0192924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:54.0193002Z return self.act(input) 2025-12-04T09:41:54.0193009Z 2025-12-04T09:41:54.0193109Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0193307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0193371Z res = mod(**inputs) 2025-12-04T09:41:54.0193626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0193702Z outputs = self.electra( 2025-12-04T09:41:54.0193951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0194019Z hidden_states = self.encoder( 2025-12-04T09:41:54.0194275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0194346Z layer_outputs = layer_module( 2025-12-04T09:41:54.0194567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0194644Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0194878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0194953Z return func(*args, **kwargs) 2025-12-04T09:41:54.0195201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0195281Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0195537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0195611Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0195900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:54.0196034Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:54.0196315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:54.0196434Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0196439Z 2025-12-04T09:41:54.0196544Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0196753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0196818Z res = mod(**inputs) 2025-12-04T09:41:54.0197081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0197157Z outputs = self.electra( 2025-12-04T09:41:54.0197419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0197522Z hidden_states = self.encoder( 2025-12-04T09:41:54.0197786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0197857Z layer_outputs = layer_module( 2025-12-04T09:41:54.0198086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0198164Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0198407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0198484Z return func(*args, **kwargs) 2025-12-04T09:41:54.0198742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0198830Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0199070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0199141Z return func(*args, **kwargs) 2025-12-04T09:41:54.0199408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0199483Z self_outputs = self.self( 2025-12-04T09:41:54.0199741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0199822Z return func(*args, **kwargs) 2025-12-04T09:41:54.0200112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:54.0200207Z query_layer = self.query(hidden_states) 2025-12-04T09:41:54.0200211Z 2025-12-04T09:41:54.0200324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0200623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0200705Z res = mod(**inputs) 2025-12-04T09:41:54.0201015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0201106Z outputs = self.electra( 2025-12-04T09:41:54.0201403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0201482Z hidden_states = self.encoder( 2025-12-04T09:41:54.0201782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0201859Z layer_outputs = layer_module( 2025-12-04T09:41:54.0202107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0202211Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0202463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0202566Z return func(*args, **kwargs) 2025-12-04T09:41:54.0202878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0202962Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0203251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0203328Z return func(*args, **kwargs) 2025-12-04T09:41:54.0203628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0203716Z self_outputs = self.self( 2025-12-04T09:41:54.0203992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0204105Z return func(*args, **kwargs) 2025-12-04T09:41:54.0204409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:54.0204498Z key_layer = self.key(current_states) 2025-12-04T09:41:54.0204502Z 2025-12-04T09:41:54.0204630Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0204852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0204932Z res = mod(**inputs) 2025-12-04T09:41:54.0205233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0205311Z outputs = self.electra( 2025-12-04T09:41:54.0205616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0205697Z hidden_states = self.encoder( 2025-12-04T09:41:54.0206067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0206157Z layer_outputs = layer_module( 2025-12-04T09:41:54.0206410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0206506Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0206781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0206858Z return func(*args, **kwargs) 2025-12-04T09:41:54.0207167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0207259Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0207532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0207616Z return func(*args, **kwargs) 2025-12-04T09:41:54.0207906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0207994Z self_outputs = self.self( 2025-12-04T09:41:54.0208266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0208341Z return func(*args, **kwargs) 2025-12-04T09:41:54.0208646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:54.0208736Z value_layer = self.value(current_states) 2025-12-04T09:41:54.0208740Z 2025-12-04T09:41:54.0208837Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0208927Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0209044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0209274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0209347Z res = mod(**inputs) 2025-12-04T09:41:54.0209662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0209782Z outputs = self.electra( 2025-12-04T09:41:54.0210070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0210165Z hidden_states = self.encoder( 2025-12-04T09:41:54.0210419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0210489Z layer_outputs = layer_module( 2025-12-04T09:41:54.0210716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0210818Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0211058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0211134Z return func(*args, **kwargs) 2025-12-04T09:41:54.0211394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0211484Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0211727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0211794Z return func(*args, **kwargs) 2025-12-04T09:41:54.0212063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:54.0212197Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:54.0212463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:54.0212550Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0212555Z 2025-12-04T09:41:54.0212660Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0212869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0212932Z res = mod(**inputs) 2025-12-04T09:41:54.0213195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0213271Z outputs = self.electra( 2025-12-04T09:41:54.0213533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0213610Z hidden_states = self.encoder( 2025-12-04T09:41:54.0213873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0213945Z layer_outputs = layer_module( 2025-12-04T09:41:54.0214174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0214253Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0214496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0214572Z return func(*args, **kwargs) 2025-12-04T09:41:54.0214832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0214922Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0215176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0215251Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0215554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0215693Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0215986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:54.0216071Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0216074Z 2025-12-04T09:41:54.0216177Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0216380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0216444Z res = mod(**inputs) 2025-12-04T09:41:54.0216715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0216781Z outputs = self.electra( 2025-12-04T09:41:54.0217060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0217137Z hidden_states = self.encoder( 2025-12-04T09:41:54.0217409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0217481Z layer_outputs = layer_module( 2025-12-04T09:41:54.0217711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0217788Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0218038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0218106Z return func(*args, **kwargs) 2025-12-04T09:41:54.0218363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0218454Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0218710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0218788Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0219090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0219211Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0219477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:54.0219590Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:54.0219807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:54.0219886Z return self.act(input) 2025-12-04T09:41:54.0219891Z 2025-12-04T09:41:54.0219994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0220198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0220264Z res = mod(**inputs) 2025-12-04T09:41:54.0220539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0220614Z outputs = self.electra( 2025-12-04T09:41:54.0220873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0220945Z hidden_states = self.encoder( 2025-12-04T09:41:54.0221222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0221289Z layer_outputs = layer_module( 2025-12-04T09:41:54.0221509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0221587Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0221822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0221917Z return func(*args, **kwargs) 2025-12-04T09:41:54.0222207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0222301Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0222560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0222636Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0222932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:54.0223066Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:54.0223339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:54.0223430Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0223434Z 2025-12-04T09:41:54.0223540Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0223745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0223809Z res = mod(**inputs) 2025-12-04T09:41:54.0224068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0224145Z outputs = self.electra( 2025-12-04T09:41:54.0224399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0224475Z hidden_states = self.encoder( 2025-12-04T09:41:54.0224732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0224804Z layer_outputs = layer_module( 2025-12-04T09:41:54.0225028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0225109Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0225348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0225426Z return func(*args, **kwargs) 2025-12-04T09:41:54.0225684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0225775Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0226013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0226082Z return func(*args, **kwargs) 2025-12-04T09:41:54.0226346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0226419Z self_outputs = self.self( 2025-12-04T09:41:54.0226657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0226733Z return func(*args, **kwargs) 2025-12-04T09:41:54.0226988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:54.0227074Z query_layer = self.query(hidden_states) 2025-12-04T09:41:54.0227078Z 2025-12-04T09:41:54.0227179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0227373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0227448Z res = mod(**inputs) 2025-12-04T09:41:54.0227710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0227827Z outputs = self.electra( 2025-12-04T09:41:54.0228126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0228198Z hidden_states = self.encoder( 2025-12-04T09:41:54.0228461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0228531Z layer_outputs = layer_module( 2025-12-04T09:41:54.0228752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0228837Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0229078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0229174Z return func(*args, **kwargs) 2025-12-04T09:41:54.0229435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0229522Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0229778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0229847Z return func(*args, **kwargs) 2025-12-04T09:41:54.0230116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0230196Z self_outputs = self.self( 2025-12-04T09:41:54.0230443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0230518Z return func(*args, **kwargs) 2025-12-04T09:41:54.0230784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:54.0230866Z key_layer = self.key(current_states) 2025-12-04T09:41:54.0230871Z 2025-12-04T09:41:54.0230984Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0231188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0231271Z res = mod(**inputs) 2025-12-04T09:41:54.0231534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0231602Z outputs = self.electra( 2025-12-04T09:41:54.0231866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0231934Z hidden_states = self.encoder( 2025-12-04T09:41:54.0232191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0232267Z layer_outputs = layer_module( 2025-12-04T09:41:54.0232488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0232575Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0232824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0232893Z return func(*args, **kwargs) 2025-12-04T09:41:54.0233167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0233250Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0233508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0233577Z return func(*args, **kwargs) 2025-12-04T09:41:54.0233843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0234843Z self_outputs = self.self( 2025-12-04T09:41:54.0235093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0235202Z return func(*args, **kwargs) 2025-12-04T09:41:54.0235481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:54.0235564Z value_layer = self.value(current_states) 2025-12-04T09:41:54.0235568Z 2025-12-04T09:41:54.0235663Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0235746Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0235854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0236070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0236158Z res = mod(**inputs) 2025-12-04T09:41:54.0236433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0236514Z outputs = self.electra( 2025-12-04T09:41:54.0236783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0236865Z hidden_states = self.encoder( 2025-12-04T09:41:54.0237138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0237210Z layer_outputs = layer_module( 2025-12-04T09:41:54.0237446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0237531Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0237791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0237875Z return func(*args, **kwargs) 2025-12-04T09:41:54.0238158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0238257Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0238522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0238602Z return func(*args, **kwargs) 2025-12-04T09:41:54.0238892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:54.0239030Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:54.0239324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:54.0239414Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0239418Z 2025-12-04T09:41:54.0239530Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0239751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0239820Z res = mod(**inputs) 2025-12-04T09:41:54.0240110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0240192Z outputs = self.electra( 2025-12-04T09:41:54.0240553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0240646Z hidden_states = self.encoder( 2025-12-04T09:41:54.0240930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0241007Z layer_outputs = layer_module( 2025-12-04T09:41:54.0241264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0241349Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0241651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0241760Z return func(*args, **kwargs) 2025-12-04T09:41:54.0242041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0242142Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0242419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0242498Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0242808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0242957Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0243232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:54.0243319Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0243322Z 2025-12-04T09:41:54.0243430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0243641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0243710Z res = mod(**inputs) 2025-12-04T09:41:54.0243988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0244058Z outputs = self.electra( 2025-12-04T09:41:54.0244322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0244403Z hidden_states = self.encoder( 2025-12-04T09:41:54.0244665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0244741Z layer_outputs = layer_module( 2025-12-04T09:41:54.0244979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0245059Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0245315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0245385Z return func(*args, **kwargs) 2025-12-04T09:41:54.0245650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0245744Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0246005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0246092Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0246392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0246517Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0246790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:54.0246904Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:54.0247272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:54.0247358Z return self.act(input) 2025-12-04T09:41:54.0247363Z 2025-12-04T09:41:54.0247468Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0247683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0247749Z res = mod(**inputs) 2025-12-04T09:41:54.0248018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0248147Z outputs = self.electra( 2025-12-04T09:41:54.0248489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0248563Z hidden_states = self.encoder( 2025-12-04T09:41:54.0248834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0248905Z layer_outputs = layer_module( 2025-12-04T09:41:54.0249135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0249215Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0249493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0249576Z return func(*args, **kwargs) 2025-12-04T09:41:54.0249842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0249940Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0250204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0250280Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0250589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:54.0250724Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:54.0250989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:54.0251079Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0251084Z 2025-12-04T09:41:54.0251189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0251405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0251473Z res = mod(**inputs) 2025-12-04T09:41:54.0251743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0251821Z outputs = self.electra( 2025-12-04T09:41:54.0252084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0252164Z hidden_states = self.encoder( 2025-12-04T09:41:54.0252433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0252506Z layer_outputs = layer_module( 2025-12-04T09:41:54.0252739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0252820Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0253068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0253146Z return func(*args, **kwargs) 2025-12-04T09:41:54.0253408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0253500Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0253746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0253826Z return func(*args, **kwargs) 2025-12-04T09:41:54.0254091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0254164Z self_outputs = self.self( 2025-12-04T09:41:54.0254432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0254540Z return func(*args, **kwargs) 2025-12-04T09:41:54.0254798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:54.0254887Z query_layer = self.query(hidden_states) 2025-12-04T09:41:54.0254890Z 2025-12-04T09:41:54.0254996Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0255193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0255265Z res = mod(**inputs) 2025-12-04T09:41:54.0255522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0255614Z outputs = self.electra( 2025-12-04T09:41:54.0255872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0255943Z hidden_states = self.encoder( 2025-12-04T09:41:54.0256215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0256283Z layer_outputs = layer_module( 2025-12-04T09:41:54.0256507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0256593Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0256833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0256910Z return func(*args, **kwargs) 2025-12-04T09:41:54.0257170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0257255Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0257506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0257578Z return func(*args, **kwargs) 2025-12-04T09:41:54.0257844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0257917Z self_outputs = self.self( 2025-12-04T09:41:54.0258157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0258232Z return func(*args, **kwargs) 2025-12-04T09:41:54.0258490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:54.0258572Z key_layer = self.key(current_states) 2025-12-04T09:41:54.0258575Z 2025-12-04T09:41:54.0258686Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0258883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0258956Z res = mod(**inputs) 2025-12-04T09:41:54.0259222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0259290Z outputs = self.electra( 2025-12-04T09:41:54.0259557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0259626Z hidden_states = self.encoder( 2025-12-04T09:41:54.0259926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0259995Z layer_outputs = layer_module( 2025-12-04T09:41:54.0260220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0260305Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0260563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0260662Z return func(*args, **kwargs) 2025-12-04T09:41:54.0260935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0261017Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0261266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0261334Z return func(*args, **kwargs) 2025-12-04T09:41:54.0261590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0261685Z self_outputs = self.self( 2025-12-04T09:41:54.0261931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0262001Z return func(*args, **kwargs) 2025-12-04T09:41:54.0262276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:54.0262355Z value_layer = self.value(current_states) 2025-12-04T09:41:54.0262358Z 2025-12-04T09:41:54.0262442Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0262522Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0262623Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0262830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0262892Z res = mod(**inputs) 2025-12-04T09:41:54.0263160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0263237Z outputs = self.electra( 2025-12-04T09:41:54.0263501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0263577Z hidden_states = self.encoder( 2025-12-04T09:41:54.0263843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0263912Z layer_outputs = layer_module( 2025-12-04T09:41:54.0264144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0264223Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0264481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0264550Z return func(*args, **kwargs) 2025-12-04T09:41:54.0264823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0264915Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0265169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0265241Z return func(*args, **kwargs) 2025-12-04T09:41:54.0265522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:54.0265653Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:54.0265932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:54.0266017Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0266021Z 2025-12-04T09:41:54.0266135Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0266347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0266432Z res = mod(**inputs) 2025-12-04T09:41:54.0266710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0266808Z outputs = self.electra( 2025-12-04T09:41:54.0267067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0267146Z hidden_states = self.encoder( 2025-12-04T09:41:54.0267403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0267473Z layer_outputs = layer_module( 2025-12-04T09:41:54.0267700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0267797Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0268050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0268121Z return func(*args, **kwargs) 2025-12-04T09:41:54.0268386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0268479Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0268737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0268812Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0269111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0269231Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0269500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:54.0269584Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0269588Z 2025-12-04T09:41:54.0269690Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0269899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0269965Z res = mod(**inputs) 2025-12-04T09:41:54.0270235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0270304Z outputs = self.electra( 2025-12-04T09:41:54.0270564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0270644Z hidden_states = self.encoder( 2025-12-04T09:41:54.0270903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0270974Z layer_outputs = layer_module( 2025-12-04T09:41:54.0271204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0271281Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0271535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0271602Z return func(*args, **kwargs) 2025-12-04T09:41:54.0271861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0271953Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0272209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0272295Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0272591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0272733Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0273036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:54.0273151Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:54.0273368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:54.0273445Z return self.act(input) 2025-12-04T09:41:54.0273448Z 2025-12-04T09:41:54.0273554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0273757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0273822Z res = mod(**inputs) 2025-12-04T09:41:54.0274102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0274182Z outputs = self.electra( 2025-12-04T09:41:54.0274446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0274525Z hidden_states = self.encoder( 2025-12-04T09:41:54.0274789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0274857Z layer_outputs = layer_module( 2025-12-04T09:41:54.0275087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0275164Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0275404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0275484Z return func(*args, **kwargs) 2025-12-04T09:41:54.0275744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0275837Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0276100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0276175Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0276477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:54.0276613Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:54.0276886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:54.0276966Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0276971Z 2025-12-04T09:41:54.0277073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0277279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0277341Z res = mod(**inputs) 2025-12-04T09:41:54.0277611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0277688Z outputs = self.electra( 2025-12-04T09:41:54.0277958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0278036Z hidden_states = self.encoder( 2025-12-04T09:41:54.0278304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0278376Z layer_outputs = layer_module( 2025-12-04T09:41:54.0278615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0278694Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0278968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0279077Z return func(*args, **kwargs) 2025-12-04T09:41:54.0279339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0279430Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0279679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0279751Z return func(*args, **kwargs) 2025-12-04T09:41:54.0280051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0280146Z self_outputs = self.self( 2025-12-04T09:41:54.0280515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0280598Z return func(*args, **kwargs) 2025-12-04T09:41:54.0280899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:54.0280998Z query_layer = self.query(hidden_states) 2025-12-04T09:41:54.0281002Z 2025-12-04T09:41:54.0281114Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0281340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0281419Z res = mod(**inputs) 2025-12-04T09:41:54.0281703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0281795Z outputs = self.electra( 2025-12-04T09:41:54.0282054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0282127Z hidden_states = self.encoder( 2025-12-04T09:41:54.0282394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0282467Z layer_outputs = layer_module( 2025-12-04T09:41:54.0282698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0282776Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0283041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0283118Z return func(*args, **kwargs) 2025-12-04T09:41:54.0283395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0283479Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0283745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0283814Z return func(*args, **kwargs) 2025-12-04T09:41:54.0284099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0284171Z self_outputs = self.self( 2025-12-04T09:41:54.0284429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0284506Z return func(*args, **kwargs) 2025-12-04T09:41:54.0284782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:54.0284861Z key_layer = self.key(current_states) 2025-12-04T09:41:54.0284871Z 2025-12-04T09:41:54.0284974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0285170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0285268Z res = mod(**inputs) 2025-12-04T09:41:54.0285550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0285652Z outputs = self.electra( 2025-12-04T09:41:54.0285940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0286010Z hidden_states = self.encoder( 2025-12-04T09:41:54.0286292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0286361Z layer_outputs = layer_module( 2025-12-04T09:41:54.0286588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0286693Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0286954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0287022Z return func(*args, **kwargs) 2025-12-04T09:41:54.0287300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0287379Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0287621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0287687Z return func(*args, **kwargs) 2025-12-04T09:41:54.0287937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0288012Z self_outputs = self.self( 2025-12-04T09:41:54.0288246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0288311Z return func(*args, **kwargs) 2025-12-04T09:41:54.0288569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:54.0288650Z value_layer = self.value(current_states) 2025-12-04T09:41:54.0288653Z 2025-12-04T09:41:54.0288740Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0288817Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0288916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0289114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0289176Z res = mod(**inputs) 2025-12-04T09:41:54.0289439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0289505Z outputs = self.electra( 2025-12-04T09:41:54.0289757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0289834Z hidden_states = self.encoder( 2025-12-04T09:41:54.0290084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0290152Z layer_outputs = layer_module( 2025-12-04T09:41:54.0290372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0290448Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0290690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0290755Z return func(*args, **kwargs) 2025-12-04T09:41:54.0291002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0291091Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0291323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0291407Z return func(*args, **kwargs) 2025-12-04T09:41:54.0291700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:54.0291828Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:54.0292081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:54.0292161Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0292164Z 2025-12-04T09:41:54.0292261Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0292455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0292537Z res = mod(**inputs) 2025-12-04T09:41:54.0292801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0292868Z outputs = self.electra( 2025-12-04T09:41:54.0293120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0293196Z hidden_states = self.encoder( 2025-12-04T09:41:54.0293446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0293514Z layer_outputs = layer_module( 2025-12-04T09:41:54.0293734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0293810Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0294047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0294116Z return func(*args, **kwargs) 2025-12-04T09:41:54.0294376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0294469Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0294725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0294810Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0295099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0295217Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0295479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:54.0295562Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0295565Z 2025-12-04T09:41:54.0295668Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0295871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0295936Z res = mod(**inputs) 2025-12-04T09:41:54.0296204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0296273Z outputs = self.electra( 2025-12-04T09:41:54.0296538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0296611Z hidden_states = self.encoder( 2025-12-04T09:41:54.0296859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0296935Z layer_outputs = layer_module( 2025-12-04T09:41:54.0297149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0297246Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0297517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0297586Z return func(*args, **kwargs) 2025-12-04T09:41:54.0297835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0297925Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0298173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0298257Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0298551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0298694Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0298957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:54.0299071Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:54.0299293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:54.0299364Z return self.act(input) 2025-12-04T09:41:54.0299367Z 2025-12-04T09:41:54.0299469Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0299677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0299742Z res = mod(**inputs) 2025-12-04T09:41:54.0300003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0300090Z outputs = self.electra( 2025-12-04T09:41:54.0300343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0300428Z hidden_states = self.encoder( 2025-12-04T09:41:54.0300680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0300749Z layer_outputs = layer_module( 2025-12-04T09:41:54.0300979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0301055Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0301294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0301368Z return func(*args, **kwargs) 2025-12-04T09:41:54.0301621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0301713Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0301967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0302043Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0302340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:54.0302473Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:54.0302742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:54.0302821Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0302824Z 2025-12-04T09:41:54.0302921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0303123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0303209Z res = mod(**inputs) 2025-12-04T09:41:54.0303470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0303571Z outputs = self.electra( 2025-12-04T09:41:54.0303823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0303899Z hidden_states = self.encoder( 2025-12-04T09:41:54.0304148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0304216Z layer_outputs = layer_module( 2025-12-04T09:41:54.0304436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0304530Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0304772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0304839Z return func(*args, **kwargs) 2025-12-04T09:41:54.0305090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0305178Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0305412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0305478Z return func(*args, **kwargs) 2025-12-04T09:41:54.0305738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0305805Z self_outputs = self.self( 2025-12-04T09:41:54.0306048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0306115Z return func(*args, **kwargs) 2025-12-04T09:41:54.0306365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:41:54.0306454Z query_layer = self.query(hidden_states) 2025-12-04T09:41:54.0306459Z 2025-12-04T09:41:54.0306559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0306756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0306819Z res = mod(**inputs) 2025-12-04T09:41:54.0307074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0307145Z outputs = self.electra( 2025-12-04T09:41:54.0307396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0307464Z hidden_states = self.encoder( 2025-12-04T09:41:54.0307722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0307795Z layer_outputs = layer_module( 2025-12-04T09:41:54.0308020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0308095Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0308327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0308399Z return func(*args, **kwargs) 2025-12-04T09:41:54.0308650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0308729Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0308969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0309034Z return func(*args, **kwargs) 2025-12-04T09:41:54.0309318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0309386Z self_outputs = self.self( 2025-12-04T09:41:54.0309648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0309724Z return func(*args, **kwargs) 2025-12-04T09:41:54.0309973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:41:54.0310056Z key_layer = self.key(current_states) 2025-12-04T09:41:54.0310060Z 2025-12-04T09:41:54.0310160Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0310349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0310442Z res = mod(**inputs) 2025-12-04T09:41:54.0310699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0310768Z outputs = self.electra( 2025-12-04T09:41:54.0311033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0311104Z hidden_states = self.encoder( 2025-12-04T09:41:54.0311371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0311442Z layer_outputs = layer_module( 2025-12-04T09:41:54.0311664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0311751Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0311990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0312061Z return func(*args, **kwargs) 2025-12-04T09:41:54.0312328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0312416Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0312664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0312733Z return func(*args, **kwargs) 2025-12-04T09:41:54.0313002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:41:54.0313080Z self_outputs = self.self( 2025-12-04T09:41:54.0313314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0313393Z return func(*args, **kwargs) 2025-12-04T09:41:54.0313655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:41:54.0313738Z value_layer = self.value(current_states) 2025-12-04T09:41:54.0313741Z 2025-12-04T09:41:54.0313829Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0313912Z cudagraph partition due to non gpu ops 2025-12-04T09:41:54.0314017Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0314223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0314287Z res = mod(**inputs) 2025-12-04T09:41:54.0314561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0314633Z outputs = self.electra( 2025-12-04T09:41:54.0314893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0314974Z hidden_states = self.encoder( 2025-12-04T09:41:54.0315231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0315320Z layer_outputs = layer_module( 2025-12-04T09:41:54.0315574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0315655Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0315902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0315973Z return func(*args, **kwargs) 2025-12-04T09:41:54.0316231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:41:54.0316324Z self_attention_outputs = self.attention( 2025-12-04T09:41:54.0316577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0316651Z return func(*args, **kwargs) 2025-12-04T09:41:54.0316911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:41:54.0317038Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:41:54.0317302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:41:54.0338827Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0338855Z 2025-12-04T09:41:54.0339168Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0339420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0339509Z res = mod(**inputs) 2025-12-04T09:41:54.0339825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0339942Z outputs = self.electra( 2025-12-04T09:41:54.0340236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0340328Z hidden_states = self.encoder( 2025-12-04T09:41:54.0340613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0340690Z layer_outputs = layer_module( 2025-12-04T09:41:54.0340930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0341029Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0341286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0341373Z return func(*args, **kwargs) 2025-12-04T09:41:54.0341648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0341745Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0342034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0342116Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0342412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0342545Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0342814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:41:54.0342910Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0342917Z 2025-12-04T09:41:54.0343036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0343258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0343421Z res = mod(**inputs) 2025-12-04T09:41:54.0343772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0343854Z outputs = self.electra( 2025-12-04T09:41:54.0344121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0344194Z hidden_states = self.encoder( 2025-12-04T09:41:54.0344531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0344605Z layer_outputs = layer_module( 2025-12-04T09:41:54.0344835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0344957Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0345203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0345286Z return func(*args, **kwargs) 2025-12-04T09:41:54.0345549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0345634Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0345893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0345982Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0346273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:41:54.0346401Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:41:54.0346665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:41:54.0346781Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:41:54.0347004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:54.0347275Z return self.act(input) 2025-12-04T09:41:54.0347281Z 2025-12-04T09:41:54.0347399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0347612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0347678Z res = mod(**inputs) 2025-12-04T09:41:54.0347954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:41:54.0348025Z outputs = self.electra( 2025-12-04T09:41:54.0348291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:41:54.0348375Z hidden_states = self.encoder( 2025-12-04T09:41:54.0348640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:41:54.0348721Z layer_outputs = layer_module( 2025-12-04T09:41:54.0348947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:54.0349030Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:54.0349287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:54.0349362Z return func(*args, **kwargs) 2025-12-04T09:41:54.0349622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:41:54.0349716Z layer_output = apply_chunking_to_forward( 2025-12-04T09:41:54.0349977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:41:54.0350129Z return forward_fn(*input_tensors) 2025-12-04T09:41:54.0350468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:41:54.0350608Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:41:54.0350875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:41:54.0350957Z hidden_states = self.dense(hidden_states) 2025-12-04T09:41:54.0350961Z 2025-12-04T09:41:54.0351080Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0351280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0351375Z res = mod(**inputs) 2025-12-04T09:41:54.0351646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1550, in forward 2025-12-04T09:41:54.0351831Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-12-04T09:41:54.0352093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 630, in forward 2025-12-04T09:41:54.0352241Z hidden_states = self.dense(generator_hidden_states) 2025-12-04T09:41:54.0352249Z 2025-12-04T09:41:54.0352359Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0352559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0352624Z res = mod(**inputs) 2025-12-04T09:41:54.0352884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1550, in forward 2025-12-04T09:41:54.0353069Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-12-04T09:41:54.0353074Z 2025-12-04T09:41:54.0353175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:54.0353376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:54.0353443Z res = mod(**inputs) 2025-12-04T09:41:54.0353706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1554, in forward 2025-12-04T09:41:54.0353786Z lm_loss = self.loss_function( 2025-12-04T09:41:54.0354030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:41:54.0354206Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:41:54.0354461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:41:54.0354657Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:41:54.0354662Z 2025-12-04T09:42:03.9397080Z Compilation time (from dynamo_timed): 17.46109962 2025-12-04T09:42:03.9492628Z pass 2025-12-04T09:42:03.9494856Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:03.9495730Z TIMING: _recursive_pre_grad_passes:0.009 _recursive_joint_graph_passes:0.46425 _recursive_post_grad_passes:0.0673 async_compile.wait:0.83298 code_gen:9.52769 inductor_compile:10.82844 backend_compile:14.35133 gc:0.00031 entire_frame_compile:17.4611 total_wall_time:17.4611 2025-12-04T09:42:03.9496710Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:8666 | FakeTensor.__torch_dispatch__:4346 | ProxyTorchDispatchMode.__torch_dispatch__:2555 2025-12-04T09:42:03.9497299Z Dynamo produced 1 graphs covering 377 ops with 0 graph breaks (0 unique) 2025-12-04T09:42:06.4004454Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:42:06.4005887Z import pynvml # type: ignore[import] 2025-12-04T09:42:09.7553794Z 2025-12-04T09:42:11.1974321Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:42:11.1974646Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:42:11.1987734Z cpu eval GPT2ForSequenceClassification 2025-12-04T09:42:11.9807764Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:12.3529956Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:12.7212278Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:19.7125453Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7126086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7132406Z res = mod(**inputs) 2025-12-04T09:42:19.7133067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7133584Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7134057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:42:19.7134497Z causal_mask = create_causal_mask( 2025-12-04T09:42:19.7134941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:42:19.7135539Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:42:19.7136125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:42:19.7136660Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:42:19.7137172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 655, in find_packed_sequence_indices 2025-12-04T09:42:19.7137750Z first_dummy_value = position_ids[:, :1] - 1 # We just need the diff on this first value to be 1 2025-12-04T09:42:19.7137996Z 2025-12-04T09:42:19.7138094Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7138376Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7138790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7139162Z res = mod(**inputs) 2025-12-04T09:42:19.7139567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1394, in forward 2025-12-04T09:42:19.7140069Z last_non_pad_token = (token_indices * non_pad_mask).argmax(-1) 2025-12-04T09:42:19.7140265Z 2025-12-04T09:42:19.7140394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7140794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7141157Z res = mod(**inputs) 2025-12-04T09:42:19.7141558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7142206Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7142639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7143066Z outputs = block( 2025-12-04T09:42:19.7143435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7143864Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7144313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7145069Z return func(*args, **kwargs) 2025-12-04T09:42:19.7145659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7146110Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7146554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7146973Z return func(*args, **kwargs) 2025-12-04T09:42:19.7147546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7148098Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7148677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7149137Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7149338Z 2025-12-04T09:42:19.7149466Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7149872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7150222Z res = mod(**inputs) 2025-12-04T09:42:19.7150627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7151062Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7151503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:42:19.7151937Z causal_mask = create_causal_mask( 2025-12-04T09:42:19.7152357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:42:19.7152893Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:42:19.7153473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:42:19.7153983Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:42:19.7154481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-12-04T09:42:19.7155009Z position_diff = torch.diff(position_ids, prepend=first_dummy_value, dim=-1) 2025-12-04T09:42:19.7155255Z 2025-12-04T09:42:19.7155345Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7155603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7155983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7156332Z res = mod(**inputs) 2025-12-04T09:42:19.7156738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7157167Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7157583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:42:19.7158002Z causal_mask = create_causal_mask( 2025-12-04T09:42:19.7158407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:42:19.7158940Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:42:19.7159498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:42:19.7159997Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:42:19.7160680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T09:42:19.7161233Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T09:42:19.7161426Z 2025-12-04T09:42:19.7161595Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7162005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7162354Z res = mod(**inputs) 2025-12-04T09:42:19.7162736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7163165Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7163585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:42:19.7164026Z causal_mask = create_causal_mask( 2025-12-04T09:42:19.7164422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:42:19.7164956Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:42:19.7165482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:42:19.7165931Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:42:19.7166389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T09:42:19.7166831Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T09:42:19.7166994Z 2025-12-04T09:42:19.7167084Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7167322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7167691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7168018Z res = mod(**inputs) 2025-12-04T09:42:19.7168386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7168785Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7169189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7169567Z outputs = block( 2025-12-04T09:42:19.7169897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7170271Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7170660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7171047Z return func(*args, **kwargs) 2025-12-04T09:42:19.7171417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7171823Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7172224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7172598Z return func(*args, **kwargs) 2025-12-04T09:42:19.7172977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7173386Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7174065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7174553Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7174747Z 2025-12-04T09:42:19.7174857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7175227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7175582Z res = mod(**inputs) 2025-12-04T09:42:19.7175980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7176375Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7176758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7177117Z outputs = block( 2025-12-04T09:42:19.7177445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7177809Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7178186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7178567Z return func(*args, **kwargs) 2025-12-04T09:42:19.7178937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7179334Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7179718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7180079Z return func(*args, **kwargs) 2025-12-04T09:42:19.7180453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7180850Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7181217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7181627Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7181823Z 2025-12-04T09:42:19.7181930Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7182286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7182594Z res = mod(**inputs) 2025-12-04T09:42:19.7182951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7183341Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7183717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7184088Z outputs = block( 2025-12-04T09:42:19.7184407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7184773Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7185142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7185512Z return func(*args, **kwargs) 2025-12-04T09:42:19.7185877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7186285Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7186682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7187073Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7187443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7187825Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7188000Z 2025-12-04T09:42:19.7188107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7188473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7188809Z res = mod(**inputs) 2025-12-04T09:42:19.7189154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7189564Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7189981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7190347Z outputs = block( 2025-12-04T09:42:19.7190664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7191026Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7191410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7191785Z return func(*args, **kwargs) 2025-12-04T09:42:19.7192161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7192603Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7193019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7193484Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7193852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7194329Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7194569Z 2025-12-04T09:42:19.7194686Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7195053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7195385Z res = mod(**inputs) 2025-12-04T09:42:19.7195758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7196158Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7196560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7196945Z outputs = block( 2025-12-04T09:42:19.7197277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7197644Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7198037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7198422Z return func(*args, **kwargs) 2025-12-04T09:42:19.7198795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7199246Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7199707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7200150Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7200630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7201109Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7201304Z 2025-12-04T09:42:19.7201434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7201834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7202156Z res = mod(**inputs) 2025-12-04T09:42:19.7202526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7202934Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7203332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7203752Z outputs = block( 2025-12-04T09:42:19.7204083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7204510Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7204900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7205283Z return func(*args, **kwargs) 2025-12-04T09:42:19.7205707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7206109Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7206512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7206925Z return func(*args, **kwargs) 2025-12-04T09:42:19.7207307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7207812Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7208293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7208698Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7208877Z 2025-12-04T09:42:19.7208972Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7209219Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7209613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7209960Z res = mod(**inputs) 2025-12-04T09:42:19.7210342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7210767Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7211170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7211551Z outputs = block( 2025-12-04T09:42:19.7211895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7212290Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7212701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7213124Z return func(*args, **kwargs) 2025-12-04T09:42:19.7213550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7213971Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7214397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7214805Z return func(*args, **kwargs) 2025-12-04T09:42:19.7215196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7215636Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7216120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7216655Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7216853Z 2025-12-04T09:42:19.7216967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7217358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7217712Z res = mod(**inputs) 2025-12-04T09:42:19.7218107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7218548Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7218971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7219412Z outputs = block( 2025-12-04T09:42:19.7219755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7220127Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7220516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7220895Z return func(*args, **kwargs) 2025-12-04T09:42:19.7221262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7221684Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7222077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7222452Z return func(*args, **kwargs) 2025-12-04T09:42:19.7222830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7223228Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7223593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7223994Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7224176Z 2025-12-04T09:42:19.7224284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7224652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7224980Z res = mod(**inputs) 2025-12-04T09:42:19.7225340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7225742Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7226143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7226514Z outputs = block( 2025-12-04T09:42:19.7226848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7227222Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7227608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7227981Z return func(*args, **kwargs) 2025-12-04T09:42:19.7228355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7228778Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7229189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7229593Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7229959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7230363Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7230536Z 2025-12-04T09:42:19.7230641Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7231012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7231340Z res = mod(**inputs) 2025-12-04T09:42:19.7231703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7232099Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7232493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7232891Z outputs = block( 2025-12-04T09:42:19.7233243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7233634Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7234020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7234401Z return func(*args, **kwargs) 2025-12-04T09:42:19.7234773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7235200Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7235617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7236036Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7236397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7236868Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7237106Z 2025-12-04T09:42:19.7237220Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7237583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7237909Z res = mod(**inputs) 2025-12-04T09:42:19.7238275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7238680Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7239094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7239494Z outputs = block( 2025-12-04T09:42:19.7239841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7240237Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7240714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7241118Z return func(*args, **kwargs) 2025-12-04T09:42:19.7241528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7241964Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7242410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7242843Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7243239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7243667Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7243859Z 2025-12-04T09:42:19.7243973Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7244365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7244705Z res = mod(**inputs) 2025-12-04T09:42:19.7245088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7245511Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7245931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7246331Z outputs = block( 2025-12-04T09:42:19.7246686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7247268Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7247752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7248213Z return func(*args, **kwargs) 2025-12-04T09:42:19.7248615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7249051Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7249433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7249806Z return func(*args, **kwargs) 2025-12-04T09:42:19.7250169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7250697Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7251153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7251548Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7251717Z 2025-12-04T09:42:19.7251808Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7252041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7252403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7252725Z res = mod(**inputs) 2025-12-04T09:42:19.7253081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7253471Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7253858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7254226Z outputs = block( 2025-12-04T09:42:19.7254547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7254904Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7255282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7255652Z return func(*args, **kwargs) 2025-12-04T09:42:19.7256008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7256399Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7256787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7257156Z return func(*args, **kwargs) 2025-12-04T09:42:19.7257512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7257959Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7258409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7258888Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7259078Z 2025-12-04T09:42:19.7259183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7259548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7259877Z res = mod(**inputs) 2025-12-04T09:42:19.7260242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7260637Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7261030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7261403Z outputs = block( 2025-12-04T09:42:19.7261738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7262135Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7262514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7262889Z return func(*args, **kwargs) 2025-12-04T09:42:19.7263268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7263669Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7264063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7264462Z return func(*args, **kwargs) 2025-12-04T09:42:19.7264823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7265210Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7265562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7265960Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7266136Z 2025-12-04T09:42:19.7266241Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7266603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7266914Z res = mod(**inputs) 2025-12-04T09:42:19.7267272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7267659Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7268046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7268411Z outputs = block( 2025-12-04T09:42:19.7268736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7269103Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7269474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7269850Z return func(*args, **kwargs) 2025-12-04T09:42:19.7270215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7270621Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7271017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7271407Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7271771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7272167Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7272334Z 2025-12-04T09:42:19.7272441Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7272800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7273115Z res = mod(**inputs) 2025-12-04T09:42:19.7273464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7273854Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7274240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7274606Z outputs = block( 2025-12-04T09:42:19.7274916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7275352Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7275752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7276110Z return func(*args, **kwargs) 2025-12-04T09:42:19.7276474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7276880Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7277282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7277662Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7278013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7278478Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7278707Z 2025-12-04T09:42:19.7278828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7279202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7279536Z res = mod(**inputs) 2025-12-04T09:42:19.7279908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7280396Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7280850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7281263Z outputs = block( 2025-12-04T09:42:19.7281626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7282014Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7282431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7282808Z return func(*args, **kwargs) 2025-12-04T09:42:19.7283170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7283575Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7283983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7284372Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7284726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7285133Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7285312Z 2025-12-04T09:42:19.7285416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7285773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7286086Z res = mod(**inputs) 2025-12-04T09:42:19.7286444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7286838Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7287214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7287579Z outputs = block( 2025-12-04T09:42:19.7287896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7288258Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7288636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7289020Z return func(*args, **kwargs) 2025-12-04T09:42:19.7289396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:42:19.7289853Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:42:19.7290063Z 2025-12-04T09:42:19.7290177Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7290560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7290905Z res = mod(**inputs) 2025-12-04T09:42:19.7291247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7291630Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7292017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7292439Z outputs = block( 2025-12-04T09:42:19.7292792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7293188Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7293607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7294007Z return func(*args, **kwargs) 2025-12-04T09:42:19.7294408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7294846Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7295271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7295672Z return func(*args, **kwargs) 2025-12-04T09:42:19.7296073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7296615Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7297123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7297559Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7297755Z 2025-12-04T09:42:19.7297848Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7298125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7298519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7298873Z res = mod(**inputs) 2025-12-04T09:42:19.7299271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7299706Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7300131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7300541Z outputs = block( 2025-12-04T09:42:19.7300908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7301315Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7301732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7302141Z return func(*args, **kwargs) 2025-12-04T09:42:19.7302535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7302937Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7303342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7303730Z return func(*args, **kwargs) 2025-12-04T09:42:19.7304103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7304534Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7305033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7305533Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7305720Z 2025-12-04T09:42:19.7305831Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7306210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7306543Z res = mod(**inputs) 2025-12-04T09:42:19.7306917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7307344Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7307744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7308133Z outputs = block( 2025-12-04T09:42:19.7308468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7308848Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7309240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7309624Z return func(*args, **kwargs) 2025-12-04T09:42:19.7309996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7310400Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7310798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7311185Z return func(*args, **kwargs) 2025-12-04T09:42:19.7311559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7311959Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7312333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7312738Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7312920Z 2025-12-04T09:42:19.7313027Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7313406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7313730Z res = mod(**inputs) 2025-12-04T09:42:19.7314093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7314499Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7314898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7315273Z outputs = block( 2025-12-04T09:42:19.7315648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7316026Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7316417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7316797Z return func(*args, **kwargs) 2025-12-04T09:42:19.7317179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7317601Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7318022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7318420Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7318819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7319285Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7319472Z 2025-12-04T09:42:19.7319585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7319977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7320319Z res = mod(**inputs) 2025-12-04T09:42:19.7320784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7321212Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7321633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7322071Z outputs = block( 2025-12-04T09:42:19.7322423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7322825Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7323244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7323651Z return func(*args, **kwargs) 2025-12-04T09:42:19.7324045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7324493Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7324937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7325361Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7325738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7326237Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7326490Z 2025-12-04T09:42:19.7326615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7327002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7327351Z res = mod(**inputs) 2025-12-04T09:42:19.7327738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7328160Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7328573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7328980Z outputs = block( 2025-12-04T09:42:19.7329334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7329727Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7330133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7330539Z return func(*args, **kwargs) 2025-12-04T09:42:19.7330943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7331379Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7331819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7332244Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7332639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7333047Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7333240Z 2025-12-04T09:42:19.7333378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7333767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7334153Z res = mod(**inputs) 2025-12-04T09:42:19.7334535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7334958Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7335375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7335765Z outputs = block( 2025-12-04T09:42:19.7336119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7336532Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7336941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7337338Z return func(*args, **kwargs) 2025-12-04T09:42:19.7337742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7338173Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7338587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7338992Z return func(*args, **kwargs) 2025-12-04T09:42:19.7339389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7339920Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7340414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7340846Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7341032Z 2025-12-04T09:42:19.7341128Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7341393Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7341776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7342124Z res = mod(**inputs) 2025-12-04T09:42:19.7342508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7342923Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7343342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7343738Z outputs = block( 2025-12-04T09:42:19.7344088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7344474Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7344882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7345291Z return func(*args, **kwargs) 2025-12-04T09:42:19.7345680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7346106Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7346524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7346925Z return func(*args, **kwargs) 2025-12-04T09:42:19.7347492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7347923Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7348410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7349023Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7349217Z 2025-12-04T09:42:19.7349384Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7349779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7350125Z res = mod(**inputs) 2025-12-04T09:42:19.7350505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7350932Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7351351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7351775Z outputs = block( 2025-12-04T09:42:19.7352119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7352523Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7352935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7353326Z return func(*args, **kwargs) 2025-12-04T09:42:19.7353721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7354157Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7354578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7355011Z return func(*args, **kwargs) 2025-12-04T09:42:19.7355404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7355823Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7356212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7356639Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7356837Z 2025-12-04T09:42:19.7356950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7357341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7357676Z res = mod(**inputs) 2025-12-04T09:42:19.7358058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7358483Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7358904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7359297Z outputs = block( 2025-12-04T09:42:19.7359652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7360051Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7360535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7360970Z return func(*args, **kwargs) 2025-12-04T09:42:19.7361387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7361846Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7362295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7362694Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7363058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7363468Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7363668Z 2025-12-04T09:42:19.7363774Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7364185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7364517Z res = mod(**inputs) 2025-12-04T09:42:19.7364875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7365277Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7365671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7366045Z outputs = block( 2025-12-04T09:42:19.7366372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7366757Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7367141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7367510Z return func(*args, **kwargs) 2025-12-04T09:42:19.7367884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7368304Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7368716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7369101Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7369454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7369914Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7370148Z 2025-12-04T09:42:19.7370263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7370622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7370948Z res = mod(**inputs) 2025-12-04T09:42:19.7371312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7371707Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7372108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7372486Z outputs = block( 2025-12-04T09:42:19.7372816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7373180Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7373572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7373943Z return func(*args, **kwargs) 2025-12-04T09:42:19.7374311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7374714Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7375115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7375507Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7375859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7376257Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7376438Z 2025-12-04T09:42:19.7376542Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7376900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7377211Z res = mod(**inputs) 2025-12-04T09:42:19.7377605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7378049Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7378428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7378797Z outputs = block( 2025-12-04T09:42:19.7379124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7379486Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7379863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7380257Z return func(*args, **kwargs) 2025-12-04T09:42:19.7380646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:42:19.7381063Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:42:19.7381223Z 2025-12-04T09:42:19.7381326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7381688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7382004Z res = mod(**inputs) 2025-12-04T09:42:19.7382361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7382766Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7383163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7383548Z outputs = block( 2025-12-04T09:42:19.7383865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7384227Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7384605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7384964Z return func(*args, **kwargs) 2025-12-04T09:42:19.7385333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7385728Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7386112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7386473Z return func(*args, **kwargs) 2025-12-04T09:42:19.7386838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7387328Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7387785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7388175Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7388349Z 2025-12-04T09:42:19.7388434Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7388673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7389020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7389339Z res = mod(**inputs) 2025-12-04T09:42:19.7389694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7390080Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7390461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7390832Z outputs = block( 2025-12-04T09:42:19.7391156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7391530Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7391949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7392319Z return func(*args, **kwargs) 2025-12-04T09:42:19.7392682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7393068Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7393453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7393820Z return func(*args, **kwargs) 2025-12-04T09:42:19.7394187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7394613Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7395074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7395581Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7395763Z 2025-12-04T09:42:19.7395868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7396226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7396545Z res = mod(**inputs) 2025-12-04T09:42:19.7396903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7397300Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7397699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7398082Z outputs = block( 2025-12-04T09:42:19.7398411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7398782Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7399173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7399566Z return func(*args, **kwargs) 2025-12-04T09:42:19.7399958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7400444Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7400879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7401290Z return func(*args, **kwargs) 2025-12-04T09:42:19.7401696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7402130Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7402526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7402974Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7403151Z 2025-12-04T09:42:19.7403256Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7403616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7403930Z res = mod(**inputs) 2025-12-04T09:42:19.7404286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7404688Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7405081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7405459Z outputs = block( 2025-12-04T09:42:19.7405806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7406212Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7406599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7406982Z return func(*args, **kwargs) 2025-12-04T09:42:19.7407349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7407759Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7408165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7408567Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7408927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7409340Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7409514Z 2025-12-04T09:42:19.7409623Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7409991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7410319Z res = mod(**inputs) 2025-12-04T09:42:19.7410683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7411078Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7411474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7411852Z outputs = block( 2025-12-04T09:42:19.7412184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7412554Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7412944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7413352Z return func(*args, **kwargs) 2025-12-04T09:42:19.7413740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7414187Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7414624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7415045Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7415401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7415867Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7416101Z 2025-12-04T09:42:19.7416217Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7416589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7416911Z res = mod(**inputs) 2025-12-04T09:42:19.7417275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7417673Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7418063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7418441Z outputs = block( 2025-12-04T09:42:19.7418774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7419151Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7419531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7419939Z return func(*args, **kwargs) 2025-12-04T09:42:19.7420348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7420766Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7421179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7421579Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7421948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7422364Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7422575Z 2025-12-04T09:42:19.7422688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7423084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7423431Z res = mod(**inputs) 2025-12-04T09:42:19.7423808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7424221Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7424618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7424987Z outputs = block( 2025-12-04T09:42:19.7425319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7425689Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7426072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7426447Z return func(*args, **kwargs) 2025-12-04T09:42:19.7426823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7427233Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7427624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7428016Z return func(*args, **kwargs) 2025-12-04T09:42:19.7428402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7428901Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7429362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7429769Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7429946Z 2025-12-04T09:42:19.7430031Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7430280Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7430650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7430982Z res = mod(**inputs) 2025-12-04T09:42:19.7431345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7431736Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7432129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7432511Z outputs = block( 2025-12-04T09:42:19.7432840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7433207Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7433592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7434022Z return func(*args, **kwargs) 2025-12-04T09:42:19.7434419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7434828Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7435228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7435611Z return func(*args, **kwargs) 2025-12-04T09:42:19.7435981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7436396Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7436858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7437388Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7437589Z 2025-12-04T09:42:19.7437704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7438104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7438456Z res = mod(**inputs) 2025-12-04T09:42:19.7438845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7439278Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7439700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7440106Z outputs = block( 2025-12-04T09:42:19.7440528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7440947Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7441371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7441780Z return func(*args, **kwargs) 2025-12-04T09:42:19.7442201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7442656Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7443078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7443474Z return func(*args, **kwargs) 2025-12-04T09:42:19.7443887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7444323Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7444731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7445173Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7445372Z 2025-12-04T09:42:19.7445488Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7445897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7446247Z res = mod(**inputs) 2025-12-04T09:42:19.7446647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7447182Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7447628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7448033Z outputs = block( 2025-12-04T09:42:19.7448395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7448802Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7449278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7449687Z return func(*args, **kwargs) 2025-12-04T09:42:19.7450149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7450613Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7451049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7451450Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7451815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7452229Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7452427Z 2025-12-04T09:42:19.7452533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7452895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7453216Z res = mod(**inputs) 2025-12-04T09:42:19.7453564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7453954Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7454337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7454706Z outputs = block( 2025-12-04T09:42:19.7455023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7455383Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7455759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7456126Z return func(*args, **kwargs) 2025-12-04T09:42:19.7456484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7456896Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7457294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7457670Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7458020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7458466Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7458692Z 2025-12-04T09:42:19.7458805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7459155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7459475Z res = mod(**inputs) 2025-12-04T09:42:19.7459828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7460224Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7460608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7460984Z outputs = block( 2025-12-04T09:42:19.7461316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7461678Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7462067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7462440Z return func(*args, **kwargs) 2025-12-04T09:42:19.7462807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7463236Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7463670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7464066Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7464420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7464821Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7464998Z 2025-12-04T09:42:19.7465101Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7465461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7465801Z res = mod(**inputs) 2025-12-04T09:42:19.7466156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7466545Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7466932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7467288Z outputs = block( 2025-12-04T09:42:19.7467608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7467970Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7468348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7468720Z return func(*args, **kwargs) 2025-12-04T09:42:19.7469086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:42:19.7469501Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:42:19.7469666Z 2025-12-04T09:42:19.7469769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7470115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7470427Z res = mod(**inputs) 2025-12-04T09:42:19.7470761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7471144Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7471515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7471875Z outputs = block( 2025-12-04T09:42:19.7472189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7472548Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7472920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7473285Z return func(*args, **kwargs) 2025-12-04T09:42:19.7473653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7474030Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7474402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7474750Z return func(*args, **kwargs) 2025-12-04T09:42:19.7475103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7475571Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7476014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7476393Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7476590Z 2025-12-04T09:42:19.7476671Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7476904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7477273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7477584Z res = mod(**inputs) 2025-12-04T09:42:19.7477927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7478310Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7478681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7479042Z outputs = block( 2025-12-04T09:42:19.7479377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7479730Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7480088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7480538Z return func(*args, **kwargs) 2025-12-04T09:42:19.7480929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7481377Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7481823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7482240Z return func(*args, **kwargs) 2025-12-04T09:42:19.7482626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7483047Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7483496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7483982Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7484163Z 2025-12-04T09:42:19.7484279Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7484633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7484966Z res = mod(**inputs) 2025-12-04T09:42:19.7485311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7485694Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7486086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7486453Z outputs = block( 2025-12-04T09:42:19.7486774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7487128Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7487510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7487888Z return func(*args, **kwargs) 2025-12-04T09:42:19.7488249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7488642Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7489030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7489396Z return func(*args, **kwargs) 2025-12-04T09:42:19.7489756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7490160Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7490528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7490956Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7491137Z 2025-12-04T09:42:19.7491275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7491645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7491975Z res = mod(**inputs) 2025-12-04T09:42:19.7492322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7492714Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7493102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7493499Z outputs = block( 2025-12-04T09:42:19.7493828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7494199Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7494587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7494963Z return func(*args, **kwargs) 2025-12-04T09:42:19.7495327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7495733Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7496138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7496525Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7496889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7497297Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7497472Z 2025-12-04T09:42:19.7497586Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7497962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7498279Z res = mod(**inputs) 2025-12-04T09:42:19.7498636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7499021Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7499407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7499776Z outputs = block( 2025-12-04T09:42:19.7500126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7500517Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7500903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7501286Z return func(*args, **kwargs) 2025-12-04T09:42:19.7501663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7502089Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7502487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7502868Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7503223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7503688Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7503924Z 2025-12-04T09:42:19.7504042Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7504430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7504749Z res = mod(**inputs) 2025-12-04T09:42:19.7505146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7505549Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7505932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7506310Z outputs = block( 2025-12-04T09:42:19.7506642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7507009Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7507390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7507796Z return func(*args, **kwargs) 2025-12-04T09:42:19.7508176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7508588Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7509006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7509411Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7509777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7510171Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7510351Z 2025-12-04T09:42:19.7510457Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7510821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7511140Z res = mod(**inputs) 2025-12-04T09:42:19.7511498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7511896Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7512288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7512661Z outputs = block( 2025-12-04T09:42:19.7512991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7513359Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7513743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7514118Z return func(*args, **kwargs) 2025-12-04T09:42:19.7514499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7514914Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7515293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7515672Z return func(*args, **kwargs) 2025-12-04T09:42:19.7516051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7516552Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7517013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7517425Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7517615Z 2025-12-04T09:42:19.7517706Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7517966Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7518347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7518714Z res = mod(**inputs) 2025-12-04T09:42:19.7519139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7519560Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7519977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7520444Z outputs = block( 2025-12-04T09:42:19.7520830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7521241Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7521669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7522115Z return func(*args, **kwargs) 2025-12-04T09:42:19.7522500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7522908Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7523307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7523704Z return func(*args, **kwargs) 2025-12-04T09:42:19.7524093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7524526Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7525008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7525558Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7525742Z 2025-12-04T09:42:19.7525847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7526217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7526545Z res = mod(**inputs) 2025-12-04T09:42:19.7526902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7527313Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7527728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7528123Z outputs = block( 2025-12-04T09:42:19.7528468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7528861Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7529270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7529668Z return func(*args, **kwargs) 2025-12-04T09:42:19.7530055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7530478Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7530894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7531286Z return func(*args, **kwargs) 2025-12-04T09:42:19.7531687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7531785Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7532022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7532153Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7532157Z 2025-12-04T09:42:19.7532276Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7532514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7532584Z res = mod(**inputs) 2025-12-04T09:42:19.7532894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7532988Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7533264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7533335Z outputs = block( 2025-12-04T09:42:19.7533579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7533677Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7533957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7534033Z return func(*args, **kwargs) 2025-12-04T09:42:19.7534306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7534423Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7534696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7534784Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7535019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7535155Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7535159Z 2025-12-04T09:42:19.7535273Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7535498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7535569Z res = mod(**inputs) 2025-12-04T09:42:19.7535838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7535939Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7536202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7536269Z outputs = block( 2025-12-04T09:42:19.7536514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7536597Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7536873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7536950Z return func(*args, **kwargs) 2025-12-04T09:42:19.7537214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7537337Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7537603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7537697Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7537929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7538122Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7538126Z 2025-12-04T09:42:19.7538249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7538462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7538533Z res = mod(**inputs) 2025-12-04T09:42:19.7538819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7538933Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7539245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7539319Z outputs = block( 2025-12-04T09:42:19.7539562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7539656Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7539924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7540007Z return func(*args, **kwargs) 2025-12-04T09:42:19.7540277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7540411Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7540697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7540793Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7541038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7541176Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7541179Z 2025-12-04T09:42:19.7541294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7541521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7541592Z res = mod(**inputs) 2025-12-04T09:42:19.7541871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7541971Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7542245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7542322Z outputs = block( 2025-12-04T09:42:19.7542570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7542658Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7542936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7543012Z return func(*args, **kwargs) 2025-12-04T09:42:19.7543289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:42:19.7543413Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:42:19.7543419Z 2025-12-04T09:42:19.7543532Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7543757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7543830Z res = mod(**inputs) 2025-12-04T09:42:19.7544114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7544211Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7544498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7544568Z outputs = block( 2025-12-04T09:42:19.7544823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7544910Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7545183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7545262Z return func(*args, **kwargs) 2025-12-04T09:42:19.7545546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7545671Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7545967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7546045Z return func(*args, **kwargs) 2025-12-04T09:42:19.7546287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7546472Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7546693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7546831Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7546834Z 2025-12-04T09:42:19.7546921Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7547034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7547334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7547417Z res = mod(**inputs) 2025-12-04T09:42:19.7547665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7547747Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7547999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7548063Z outputs = block( 2025-12-04T09:42:19.7548292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7548372Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7548610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7548688Z return func(*args, **kwargs) 2025-12-04T09:42:19.7548932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7549021Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7549267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7549337Z return func(*args, **kwargs) 2025-12-04T09:42:19.7549588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7549686Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7549977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7550118Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7550123Z 2025-12-04T09:42:19.7550226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7550436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7550502Z res = mod(**inputs) 2025-12-04T09:42:19.7550750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7550839Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7551081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7551144Z outputs = block( 2025-12-04T09:42:19.7551371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7551451Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7551696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7551831Z return func(*args, **kwargs) 2025-12-04T09:42:19.7552138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7552236Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7552472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7552541Z return func(*args, **kwargs) 2025-12-04T09:42:19.7552793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7552876Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7553125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7553240Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7553246Z 2025-12-04T09:42:19.7553348Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7553554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7553617Z res = mod(**inputs) 2025-12-04T09:42:19.7553867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7553949Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7554188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7554258Z outputs = block( 2025-12-04T09:42:19.7554475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7554557Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7554803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7554871Z return func(*args, **kwargs) 2025-12-04T09:42:19.7555122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7555224Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7555465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7555554Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7555764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7555876Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7555889Z 2025-12-04T09:42:19.7555990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7556187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7556256Z res = mod(**inputs) 2025-12-04T09:42:19.7556506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7556589Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7556843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7556907Z outputs = block( 2025-12-04T09:42:19.7557136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7557217Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7557461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7557537Z return func(*args, **kwargs) 2025-12-04T09:42:19.7557804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7557943Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7558200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7558279Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7558501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7558683Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7558686Z 2025-12-04T09:42:19.7558790Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7559021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7559090Z res = mod(**inputs) 2025-12-04T09:42:19.7559347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7559432Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7559678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7559751Z outputs = block( 2025-12-04T09:42:19.7559975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7560055Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7560306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7560427Z return func(*args, **kwargs) 2025-12-04T09:42:19.7560718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7560836Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7561110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7561212Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7561451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7561645Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7561653Z 2025-12-04T09:42:19.7561810Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7562017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7562092Z res = mod(**inputs) 2025-12-04T09:42:19.7562378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7562461Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7562715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7562778Z outputs = block( 2025-12-04T09:42:19.7563005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7563083Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7563319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7563395Z return func(*args, **kwargs) 2025-12-04T09:42:19.7563637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7563725Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7563967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7564077Z return func(*args, **kwargs) 2025-12-04T09:42:19.7564363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7564570Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7564814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7564951Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7564955Z 2025-12-04T09:42:19.7565048Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7565171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7565410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7565481Z res = mod(**inputs) 2025-12-04T09:42:19.7565768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7565862Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7566136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7566214Z outputs = block( 2025-12-04T09:42:19.7566460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7566555Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7566824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7566900Z return func(*args, **kwargs) 2025-12-04T09:42:19.7567185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7567285Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7567560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7567636Z return func(*args, **kwargs) 2025-12-04T09:42:19.7567909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7568026Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7568353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7568496Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7568508Z 2025-12-04T09:42:19.7568626Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7568853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7568933Z res = mod(**inputs) 2025-12-04T09:42:19.7569210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7569305Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7569587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7569657Z outputs = block( 2025-12-04T09:42:19.7569910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7569997Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7570262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7570346Z return func(*args, **kwargs) 2025-12-04T09:42:19.7570670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7570789Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7571096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7571178Z return func(*args, **kwargs) 2025-12-04T09:42:19.7571431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7571511Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7571725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7571846Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7571866Z 2025-12-04T09:42:19.7571973Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7572182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7572251Z res = mod(**inputs) 2025-12-04T09:42:19.7572508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7572600Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7572849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7572915Z outputs = block( 2025-12-04T09:42:19.7573148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7573227Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7573479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7573549Z return func(*args, **kwargs) 2025-12-04T09:42:19.7573802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7573915Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7574171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7574254Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7574485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7574601Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7574604Z 2025-12-04T09:42:19.7574719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7574920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7574987Z res = mod(**inputs) 2025-12-04T09:42:19.7575248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7575334Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7575593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7575658Z outputs = block( 2025-12-04T09:42:19.7575882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7575968Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7576212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7576281Z return func(*args, **kwargs) 2025-12-04T09:42:19.7576537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7576642Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7576934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7577011Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7577286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7577474Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7577478Z 2025-12-04T09:42:19.7577581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7577782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7577846Z res = mod(**inputs) 2025-12-04T09:42:19.7578092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7578202Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7578457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7578523Z outputs = block( 2025-12-04T09:42:19.7578754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7578833Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7579086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7579157Z return func(*args, **kwargs) 2025-12-04T09:42:19.7579407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7579529Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7579774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7579863Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7580089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7580203Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7580207Z 2025-12-04T09:42:19.7580316Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7580510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7580572Z res = mod(**inputs) 2025-12-04T09:42:19.7580827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7580909Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7581158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7581222Z outputs = block( 2025-12-04T09:42:19.7581442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7581530Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7581767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7581835Z return func(*args, **kwargs) 2025-12-04T09:42:19.7582083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:42:19.7582189Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:42:19.7582193Z 2025-12-04T09:42:19.7582300Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7582495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7582557Z res = mod(**inputs) 2025-12-04T09:42:19.7582812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7582914Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7583193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7583259Z outputs = block( 2025-12-04T09:42:19.7583479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7583563Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7583798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7583865Z return func(*args, **kwargs) 2025-12-04T09:42:19.7584169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7584257Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7584506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7584578Z return func(*args, **kwargs) 2025-12-04T09:42:19.7584826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:42:19.7585022Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:42:19.7585243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7585369Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7585372Z 2025-12-04T09:42:19.7585458Z cudagraph partition due to non gpu ops 2025-12-04T09:42:19.7585561Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7585767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7585834Z res = mod(**inputs) 2025-12-04T09:42:19.7586088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7586188Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7586429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7586497Z outputs = block( 2025-12-04T09:42:19.7586716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7586795Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7587038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7587108Z return func(*args, **kwargs) 2025-12-04T09:42:19.7587348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7587446Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7587683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7587759Z return func(*args, **kwargs) 2025-12-04T09:42:19.7587998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:42:19.7588094Z attn_output, attn_weights = attention_interface( 2025-12-04T09:42:19.7588388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:42:19.7588515Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:42:19.7588519Z 2025-12-04T09:42:19.7588624Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7588841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7588905Z res = mod(**inputs) 2025-12-04T09:42:19.7589188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7589273Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7589519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7589587Z outputs = block( 2025-12-04T09:42:19.7589804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7589889Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7590146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7590215Z return func(*args, **kwargs) 2025-12-04T09:42:19.7590464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:42:19.7590552Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:42:19.7590797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7590864Z return func(*args, **kwargs) 2025-12-04T09:42:19.7591103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:42:19.7591190Z attn_output = self.c_proj(attn_output) 2025-12-04T09:42:19.7591403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7591518Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7591522Z 2025-12-04T09:42:19.7591627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7591823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7591891Z res = mod(**inputs) 2025-12-04T09:42:19.7592137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7592219Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7592466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7592529Z outputs = block( 2025-12-04T09:42:19.7592745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7592833Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7593076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7593157Z return func(*args, **kwargs) 2025-12-04T09:42:19.7593408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7593521Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7593792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:42:19.7593877Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:42:19.7594117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7594244Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7594249Z 2025-12-04T09:42:19.7594359Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7594581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7594674Z res = mod(**inputs) 2025-12-04T09:42:19.7594939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7595058Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7595307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7595383Z outputs = block( 2025-12-04T09:42:19.7595619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7595703Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7595966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7596068Z return func(*args, **kwargs) 2025-12-04T09:42:19.7596342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7596483Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7596751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:42:19.7596845Z hidden_states = self.act(hidden_states) 2025-12-04T09:42:19.7597078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:19.7597266Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:19.7597277Z 2025-12-04T09:42:19.7597387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7597599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7597677Z res = mod(**inputs) 2025-12-04T09:42:19.7597948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:42:19.7598038Z transformer_outputs = self.transformer( 2025-12-04T09:42:19.7598314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:42:19.7598383Z outputs = block( 2025-12-04T09:42:19.7598628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:19.7598711Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:19.7598970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:19.7599053Z return func(*args, **kwargs) 2025-12-04T09:42:19.7599320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:42:19.7599431Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:42:19.7599704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:42:19.7599797Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:42:19.7600042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:42:19.7600167Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:42:19.7600170Z 2025-12-04T09:42:19.7600281Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7600593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7600671Z res = mod(**inputs) 2025-12-04T09:42:19.7600956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1379, in forward 2025-12-04T09:42:19.7601044Z logits = self.score(hidden_states) 2025-12-04T09:42:19.7601049Z 2025-12-04T09:42:19.7601164Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7601415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7602761Z res = mod(**inputs) 2025-12-04T09:42:19.7603046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1422, in forward 2025-12-04T09:42:19.7603210Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-12-04T09:42:19.7603215Z 2025-12-04T09:42:19.7603325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:19.7603544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:19.7603613Z res = mod(**inputs) 2025-12-04T09:42:19.7603881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1422, in forward 2025-12-04T09:42:19.7604066Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-12-04T09:42:19.7604071Z 2025-12-04T09:42:32.4250615Z Compilation time (from dynamo_timed): 18.951288743 2025-12-04T09:42:32.4250944Z pass 2025-12-04T09:42:32.4251294Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:32.4252183Z TIMING: _recursive_pre_grad_passes:0.01387 _recursive_joint_graph_passes:0.61749 _recursive_post_grad_passes:0.07092 async_compile.wait:0.76306 code_gen:9.69816 inductor_compile:10.95067 backend_compile:14.17368 gc:0.00107 entire_frame_compile:18.95129 total_wall_time:18.95129 2025-12-04T09:42:32.4253176Z STATS: call_* op count: 1126 | FakeTensorMode.__torch_dispatch__:7443 | FakeTensor.__torch_dispatch__:4239 | ProxyTorchDispatchMode.__torch_dispatch__:1681 2025-12-04T09:42:32.4253710Z Dynamo produced 2 graphs covering 1126 ops with 0 graph breaks (0 unique) 2025-12-04T09:42:34.8911660Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:42:34.8912510Z import pynvml # type: ignore[import] 2025-12-04T09:42:38.4136207Z 2025-12-04T09:42:39.2832783Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:42:39.2833091Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:42:39.2840869Z cpu eval GoogleFnet 2025-12-04T09:42:39.7282860Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:39.9283650Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:40.1302350Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:45.4089389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4093752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4099098Z res = mod(**inputs) 2025-12-04T09:42:45.4102364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4102903Z outputs = self.fnet( 2025-12-04T09:42:45.4109284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4110897Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4111362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4111795Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4112200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4112603Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4113028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4113809Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4114351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4114834Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4115240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4115683Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4115854Z 2025-12-04T09:42:45.4115981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4116398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4117092Z res = mod(**inputs) 2025-12-04T09:42:45.4117486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4117901Z outputs = self.fnet( 2025-12-04T09:42:45.4118295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4118721Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4119127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4119559Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4119977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4120560Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4121005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4121471Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4121930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4122352Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4122767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4123213Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4123381Z 2025-12-04T09:42:45.4123506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4123894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4124245Z res = mod(**inputs) 2025-12-04T09:42:45.4124630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4125037Z outputs = self.fnet( 2025-12-04T09:42:45.4125417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4125831Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4126239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4126662Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4127103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4127521Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4127941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4128457Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4128895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4129349Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4129809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4130314Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4130488Z 2025-12-04T09:42:45.4130602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4130995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4131334Z res = mod(**inputs) 2025-12-04T09:42:45.4131711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4132097Z outputs = self.fnet( 2025-12-04T09:42:45.4132487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4132874Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4133255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4133657Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4134035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4134396Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4134783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4135194Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4135593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4135987Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4136375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4136789Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4136947Z 2025-12-04T09:42:45.4137057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4137424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4137749Z res = mod(**inputs) 2025-12-04T09:42:45.4138097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4138474Z outputs = self.fnet( 2025-12-04T09:42:45.4138829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4139216Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4139589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4139989Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4140369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4140739Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4141121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4141530Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4141935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4142322Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4142711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4143133Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4143328Z 2025-12-04T09:42:45.4143450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4143906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4144235Z res = mod(**inputs) 2025-12-04T09:42:45.4144612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4145017Z outputs = self.fnet( 2025-12-04T09:42:45.4145378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4145769Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4146150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4146622Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4146999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4147546Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4147946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4148352Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4148767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4149167Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4149552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4149970Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4150138Z 2025-12-04T09:42:45.4150247Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4150615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4150938Z res = mod(**inputs) 2025-12-04T09:42:45.4151301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4151686Z outputs = self.fnet( 2025-12-04T09:42:45.4152046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4152428Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4152812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4153215Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4153591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4153951Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4154331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4154729Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4155119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4155499Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4155879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4156277Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4156437Z 2025-12-04T09:42:45.4156541Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4156895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4157216Z res = mod(**inputs) 2025-12-04T09:42:45.4157557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4157994Z outputs = self.fnet( 2025-12-04T09:42:45.4158402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4158787Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4159156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4159553Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4159925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4160288Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4160846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4161269Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4161735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4162156Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4162556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4162980Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4163141Z 2025-12-04T09:42:45.4163258Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4163626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4163947Z res = mod(**inputs) 2025-12-04T09:42:45.4164310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4164685Z outputs = self.fnet( 2025-12-04T09:42:45.4165042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4165426Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4165803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4166196Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4166562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4166926Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4167306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4167716Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4168123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4168514Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4168899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4169313Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4169470Z 2025-12-04T09:42:45.4169585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4169945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4170261Z res = mod(**inputs) 2025-12-04T09:42:45.4170613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4170989Z outputs = self.fnet( 2025-12-04T09:42:45.4171340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4171754Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4172136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4172557Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4172911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4173269Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4173644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4174031Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4174428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4174835Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4175228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4175703Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4175871Z 2025-12-04T09:42:45.4175981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4176350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4176679Z res = mod(**inputs) 2025-12-04T09:42:45.4177032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4177411Z outputs = self.fnet( 2025-12-04T09:42:45.4177772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4178164Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4178535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4178940Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4179302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4179647Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4180037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4180447Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4180847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4181242Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4181636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4182079Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4182233Z 2025-12-04T09:42:45.4182337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4182695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4183009Z res = mod(**inputs) 2025-12-04T09:42:45.4183358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4183721Z outputs = self.fnet( 2025-12-04T09:42:45.4184096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4184542Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4184939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4185368Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4185772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4186173Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4186587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4186999Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4187431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4187852Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4188241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4188663Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4188836Z 2025-12-04T09:42:45.4188952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4189308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4189637Z res = mod(**inputs) 2025-12-04T09:42:45.4189997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4190376Z outputs = self.fnet( 2025-12-04T09:42:45.4190726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 511, in forward 2025-12-04T09:42:45.4191127Z embedding_output = self.embeddings( 2025-12-04T09:42:45.4191521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 141, in forward 2025-12-04T09:42:45.4191916Z embeddings = self.projection(embeddings) 2025-12-04T09:42:45.4192064Z 2025-12-04T09:42:45.4192152Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4192412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4192796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4193139Z res = mod(**inputs) 2025-12-04T09:42:45.4193525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4193932Z outputs = self.fnet( 2025-12-04T09:42:45.4194309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4194727Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4195134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4195563Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4195957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4196354Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4196768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4197210Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4197637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4198059Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4198473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4198904Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4199078Z 2025-12-04T09:42:45.4199192Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4199580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4199927Z res = mod(**inputs) 2025-12-04T09:42:45.4200402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4200834Z outputs = self.fnet( 2025-12-04T09:42:45.4201265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4201688Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4202114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4202516Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4202892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4203255Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4204342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4204761Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4205177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4205575Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4205970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4206391Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4206552Z 2025-12-04T09:42:45.4206661Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4207033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4207365Z res = mod(**inputs) 2025-12-04T09:42:45.4207728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4208104Z outputs = self.fnet( 2025-12-04T09:42:45.4208466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4208873Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4209280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4209692Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4210071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4210446Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4210829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4211246Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4211658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4212056Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4212451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4212870Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4213028Z 2025-12-04T09:42:45.4213146Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4213519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4213834Z res = mod(**inputs) 2025-12-04T09:42:45.4214193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4214573Z outputs = self.fnet( 2025-12-04T09:42:45.4214921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4215438Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4215877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4216302Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4216679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4217058Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4217460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4217886Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4218300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4218734Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4219144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4219576Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4219742Z 2025-12-04T09:42:45.4219849Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4220209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4220555Z res = mod(**inputs) 2025-12-04T09:42:45.4220930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4221335Z outputs = self.fnet( 2025-12-04T09:42:45.4221712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4222116Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4222513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4222939Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4223319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4223682Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4224069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4224491Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4224900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4225315Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4225711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4226151Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4226551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4227071Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4227205Z 2025-12-04T09:42:45.4227314Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4227659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4227960Z res = mod(**inputs) 2025-12-04T09:42:45.4228299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4228664Z outputs = self.fnet( 2025-12-04T09:42:45.4229001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4229377Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4229743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4230155Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4230541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4230902Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4231280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4231663Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4232067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4232463Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4232894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4233333Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4233753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4234178Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4234563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4235023Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4235270Z 2025-12-04T09:42:45.4235382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4235747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4236080Z res = mod(**inputs) 2025-12-04T09:42:45.4236435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4236832Z outputs = self.fnet( 2025-12-04T09:42:45.4237204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4237594Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4237982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4238390Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4238774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4239145Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4239541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4239953Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4240472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4240924Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4241390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4241913Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4242398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4242838Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4242993Z 2025-12-04T09:42:45.4243094Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4243367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4243763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4244154Z res = mod(**inputs) 2025-12-04T09:42:45.4244548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4244996Z outputs = self.fnet( 2025-12-04T09:42:45.4245402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4245853Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4246283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4246736Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4247268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4247753Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4248181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4248618Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4249045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4249439Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4249820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4250237Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4250393Z 2025-12-04T09:42:45.4250506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4250866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4251187Z res = mod(**inputs) 2025-12-04T09:42:45.4251541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4251923Z outputs = self.fnet( 2025-12-04T09:42:45.4252272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4252657Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4253033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4253436Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4253800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4254166Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4254552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4254955Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4255360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4255762Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4256153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4256555Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4256715Z 2025-12-04T09:42:45.4256825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4257193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4257517Z res = mod(**inputs) 2025-12-04T09:42:45.4257862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4258243Z outputs = self.fnet( 2025-12-04T09:42:45.4258597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4258995Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4259413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4259809Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4260176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4260536Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4260917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4261325Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4261741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4262121Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4262510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4262915Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4263069Z 2025-12-04T09:42:45.4263173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4263528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4263842Z res = mod(**inputs) 2025-12-04T09:42:45.4264189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4264562Z outputs = self.fnet( 2025-12-04T09:42:45.4264899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4265271Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4265629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4266018Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4266389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4266754Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4267124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4267522Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4267918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4268303Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4268677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4269087Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4269240Z 2025-12-04T09:42:45.4269355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4269706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4270023Z res = mod(**inputs) 2025-12-04T09:42:45.4270373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4270748Z outputs = self.fnet( 2025-12-04T09:42:45.4271091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4271465Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4271838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4272219Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4272609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4273049Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4273424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4273797Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4274189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4274571Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4274966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4275421Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4275841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4276233Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4276371Z 2025-12-04T09:42:45.4276480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4276854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4277166Z res = mod(**inputs) 2025-12-04T09:42:45.4277519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4277882Z outputs = self.fnet( 2025-12-04T09:42:45.4278235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4278616Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4278990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4279376Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4279744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4280120Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4280571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4280983Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4281398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4281814Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4282216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4282672Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4283099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4283577Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4283952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4284408Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4284635Z 2025-12-04T09:42:45.4284751Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4285102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4285422Z res = mod(**inputs) 2025-12-04T09:42:45.4285776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4286152Z outputs = self.fnet( 2025-12-04T09:42:45.4286524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4286935Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4287310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4287695Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4288060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4288420Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4288802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4289197Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4289603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4290000Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4290414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4290861Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4291293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4291684Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4291822Z 2025-12-04T09:42:45.4291911Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4292148Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4292503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4292824Z res = mod(**inputs) 2025-12-04T09:42:45.4293170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4293542Z outputs = self.fnet( 2025-12-04T09:42:45.4293898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4294274Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4294645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4295043Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4295408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4295759Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4296142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4296547Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4297010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4297388Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4297766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4298165Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4298319Z 2025-12-04T09:42:45.4298420Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4298770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4299080Z res = mod(**inputs) 2025-12-04T09:42:45.4299423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4299782Z outputs = self.fnet( 2025-12-04T09:42:45.4300157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4300532Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4300943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4301315Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4301671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4302021Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4302383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4302793Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4303181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4303559Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4303927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4304321Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4304473Z 2025-12-04T09:42:45.4304583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4304929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4305235Z res = mod(**inputs) 2025-12-04T09:42:45.4305575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4305938Z outputs = self.fnet( 2025-12-04T09:42:45.4306273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4306647Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4307010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4307397Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4307745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4308095Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4308464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4308852Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4309245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4309632Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4310013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4310413Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4310572Z 2025-12-04T09:42:45.4310680Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4311038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4311359Z res = mod(**inputs) 2025-12-04T09:42:45.4311690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4312056Z outputs = self.fnet( 2025-12-04T09:42:45.4312397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4312759Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4313121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4313519Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4313905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4314251Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4314627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4315028Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4315424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4315799Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4316169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4316585Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4316739Z 2025-12-04T09:42:45.4316843Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4317206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4317531Z res = mod(**inputs) 2025-12-04T09:42:45.4317890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4318265Z outputs = self.fnet( 2025-12-04T09:42:45.4318628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4319075Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4319474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4319906Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4320308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4320792Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4321242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4321692Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4322108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4322514Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4322952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4323429Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4323868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4324272Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4324423Z 2025-12-04T09:42:45.4324534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4324906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4325229Z res = mod(**inputs) 2025-12-04T09:42:45.4325594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4325976Z outputs = self.fnet( 2025-12-04T09:42:45.4326337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4326765Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4327208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4327623Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4328031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4328431Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4328821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4329224Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4329631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4330025Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4330438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4330914Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4331333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4331773Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4332142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4332585Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4332811Z 2025-12-04T09:42:45.4332918Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4333277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4333595Z res = mod(**inputs) 2025-12-04T09:42:45.4333944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4334311Z outputs = self.fnet( 2025-12-04T09:42:45.4334673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4335056Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4335421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4335811Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4336173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4336528Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4336903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4337289Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4337690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4338075Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4338483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4338944Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4339387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4339782Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4339931Z 2025-12-04T09:42:45.4340015Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4340265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4340631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4340956Z res = mod(**inputs) 2025-12-04T09:42:45.4341326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4341742Z outputs = self.fnet( 2025-12-04T09:42:45.4342121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4342565Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4342969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4343395Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4343785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4344174Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4344585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4345039Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4345472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4345917Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4346300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4346697Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4346860Z 2025-12-04T09:42:45.4346968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4347484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4347878Z res = mod(**inputs) 2025-12-04T09:42:45.4348306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4348735Z outputs = self.fnet( 2025-12-04T09:42:45.4349125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4349540Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4349966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4350405Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4350809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4351203Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4351625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4352071Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4352513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4352935Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4353368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4353828Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4354001Z 2025-12-04T09:42:45.4354115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4354507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4354864Z res = mod(**inputs) 2025-12-04T09:42:45.4355251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4355654Z outputs = self.fnet( 2025-12-04T09:42:45.4356042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4356463Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4356866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4357350Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4357806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4358179Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4358564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4358984Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4359402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4359824Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4360270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4360788Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4360966Z 2025-12-04T09:42:45.4361093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4361487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4361834Z res = mod(**inputs) 2025-12-04T09:42:45.4362219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4362631Z outputs = self.fnet( 2025-12-04T09:42:45.4362987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4363383Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4363776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4364169Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4364541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4364914Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4365305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4365713Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4366124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4366518Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4366907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4367318Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4367482Z 2025-12-04T09:42:45.4367590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4367957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4368276Z res = mod(**inputs) 2025-12-04T09:42:45.4368638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4369021Z outputs = self.fnet( 2025-12-04T09:42:45.4369376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4369763Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4370130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4370517Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4370876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4371268Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4371644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4372067Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4372465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4372857Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4373262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4373712Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4374130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4374545Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4374683Z 2025-12-04T09:42:45.4374796Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4375159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4375476Z res = mod(**inputs) 2025-12-04T09:42:45.4375829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4376203Z outputs = self.fnet( 2025-12-04T09:42:45.4376547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4376928Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4377306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4377700Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4378065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4378429Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4378813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4379200Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4379602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4379995Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4380400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4380843Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4381265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4381681Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4382064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4382514Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4382751Z 2025-12-04T09:42:45.4382857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4383229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4383536Z res = mod(**inputs) 2025-12-04T09:42:45.4383886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4384252Z outputs = self.fnet( 2025-12-04T09:42:45.4384613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4385018Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4385404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4385841Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4386233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4386591Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4386983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4387387Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4387797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4388230Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4388668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4389154Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4389593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4389994Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4390142Z 2025-12-04T09:42:45.4390229Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4390458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4390810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4391123Z res = mod(**inputs) 2025-12-04T09:42:45.4391466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4391828Z outputs = self.fnet( 2025-12-04T09:42:45.4392172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4392541Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4392903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4393279Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4393637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4393989Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4394351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4394750Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4395148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4395529Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4395910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4396316Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4396471Z 2025-12-04T09:42:45.4396583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4396937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4397248Z res = mod(**inputs) 2025-12-04T09:42:45.4397599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4397977Z outputs = self.fnet( 2025-12-04T09:42:45.4398321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4398701Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4399090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4399522Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4399892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4400263Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4400729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4401161Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4401613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4402036Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4402434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4402850Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4403026Z 2025-12-04T09:42:45.4403134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4403493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4403816Z res = mod(**inputs) 2025-12-04T09:42:45.4404165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4404545Z outputs = self.fnet( 2025-12-04T09:42:45.4404897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4405267Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4405643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4406041Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4406404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4406759Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4407136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4407538Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4407933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4408321Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4408706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4409109Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4409262Z 2025-12-04T09:42:45.4409367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4409731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4410052Z res = mod(**inputs) 2025-12-04T09:42:45.4410417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4410782Z outputs = self.fnet( 2025-12-04T09:42:45.4411132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4411516Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4411882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4412279Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4412644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4413043Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4413455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4413865Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4414253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4414623Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4414995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4415392Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4415561Z 2025-12-04T09:42:45.4415673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4416015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4416331Z res = mod(**inputs) 2025-12-04T09:42:45.4416681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4417051Z outputs = self.fnet( 2025-12-04T09:42:45.4417388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4417759Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4418123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4418501Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4418861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4419216Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4419592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4419970Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4420367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4420756Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4421147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4421589Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4421997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4422382Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4422515Z 2025-12-04T09:42:45.4422617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4422967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4423277Z res = mod(**inputs) 2025-12-04T09:42:45.4423620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4423974Z outputs = self.fnet( 2025-12-04T09:42:45.4424327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4424697Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4425053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4425434Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4425799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4426150Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4426536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4426945Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4427334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4427712Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4428097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4428532Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4428935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4429346Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4429717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4430156Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4430379Z 2025-12-04T09:42:45.4430488Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4430827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4431132Z res = mod(**inputs) 2025-12-04T09:42:45.4431473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4431834Z outputs = self.fnet( 2025-12-04T09:42:45.4432166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4432534Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4432906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4433271Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4433620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4433961Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4434318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4434675Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4435051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4435417Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4435806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4436244Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4436660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4437041Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4437173Z 2025-12-04T09:42:45.4437251Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4437483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4437828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4438136Z res = mod(**inputs) 2025-12-04T09:42:45.4438470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4438833Z outputs = self.fnet( 2025-12-04T09:42:45.4439170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4439559Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4439962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4440428Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4440840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4441224Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4441640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4442074Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4442473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4442875Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4443268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4443682Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4443851Z 2025-12-04T09:42:45.4443959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4444320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4444640Z res = mod(**inputs) 2025-12-04T09:42:45.4444990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4445362Z outputs = self.fnet( 2025-12-04T09:42:45.4445717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4446103Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4446472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4446871Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4447368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4447754Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4448140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4448559Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4448975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4449365Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4449746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4450157Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4450318Z 2025-12-04T09:42:45.4450433Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4450793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4451109Z res = mod(**inputs) 2025-12-04T09:42:45.4451459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4451834Z outputs = self.fnet( 2025-12-04T09:42:45.4452173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4452556Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4452931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4453319Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4453738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4454110Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4454546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4454943Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4455350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4455740Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4456130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4456568Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4456728Z 2025-12-04T09:42:45.4456834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4457189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4457500Z res = mod(**inputs) 2025-12-04T09:42:45.4457864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4458235Z outputs = self.fnet( 2025-12-04T09:42:45.4458590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4458960Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4459335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4459726Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4460083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4460441Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4460824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4461227Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4461622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4462008Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4462395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4462805Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4462959Z 2025-12-04T09:42:45.4463063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4463420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4463740Z res = mod(**inputs) 2025-12-04T09:42:45.4464083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4464461Z outputs = self.fnet( 2025-12-04T09:42:45.4464821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4465189Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4465544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4465928Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4466284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4466643Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4467002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4467410Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4467840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4468218Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4468613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4469056Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4469470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4469849Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4469993Z 2025-12-04T09:42:45.4470116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4470474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4470800Z res = mod(**inputs) 2025-12-04T09:42:45.4471136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4471498Z outputs = self.fnet( 2025-12-04T09:42:45.4471844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4472205Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4472569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4472946Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4473301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4473648Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4474016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4474397Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4474784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4475166Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4475559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4475994Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4476392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4476805Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4477180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4477638Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4477869Z 2025-12-04T09:42:45.4477978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4478348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4478678Z res = mod(**inputs) 2025-12-04T09:42:45.4479035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4479417Z outputs = self.fnet( 2025-12-04T09:42:45.4479779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4480177Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4480660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4481138Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4481587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4481986Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4482400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4482800Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4483220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4483617Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4484035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4484526Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4484972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4485372Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4485520Z 2025-12-04T09:42:45.4485605Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4485853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4486216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4486538Z res = mod(**inputs) 2025-12-04T09:42:45.4486901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4487284Z outputs = self.fnet( 2025-12-04T09:42:45.4487641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4488038Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4488423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4488831Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4489203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4489576Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4489968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4490380Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4490793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4491197Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4491593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4492011Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4492186Z 2025-12-04T09:42:45.4492291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4492646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4493059Z res = mod(**inputs) 2025-12-04T09:42:45.4493405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4493782Z outputs = self.fnet( 2025-12-04T09:42:45.4494139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4494500Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4494866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4495270Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4495626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4496005Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4496397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4514525Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4515227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4515683Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4516115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4516688Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4516862Z 2025-12-04T09:42:45.4516980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4517350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4517693Z res = mod(**inputs) 2025-12-04T09:42:45.4518063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4518458Z outputs = self.fnet( 2025-12-04T09:42:45.4518832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4519223Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4519622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4520036Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4520272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4520448Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4520719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4520830Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4521133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4521227Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4521523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4521641Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4521648Z 2025-12-04T09:42:45.4521769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4522002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4522086Z res = mod(**inputs) 2025-12-04T09:42:45.4522388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4522466Z outputs = self.fnet( 2025-12-04T09:42:45.4522767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4522854Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4523140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4523236Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4523480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4523724Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4523990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4524146Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4524467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4524557Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4524841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4524941Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4524944Z 2025-12-04T09:42:45.4525057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4525257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4525343Z res = mod(**inputs) 2025-12-04T09:42:45.4525604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4525676Z outputs = self.fnet( 2025-12-04T09:42:45.4525927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4526009Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4526254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4526348Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4526573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4526653Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4526910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4526998Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4527270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4527362Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4527646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4527773Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4528023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4528109Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4528113Z 2025-12-04T09:42:45.4528226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4528427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4528499Z res = mod(**inputs) 2025-12-04T09:42:45.4528751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4528817Z outputs = self.fnet( 2025-12-04T09:42:45.4529077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4529150Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4529398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4529490Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4529715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4529800Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4530048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4530152Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4530446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4530527Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4530811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4530926Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4531172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4531291Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4531506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4531707Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4531718Z 2025-12-04T09:42:45.4531821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4532022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4532095Z res = mod(**inputs) 2025-12-04T09:42:45.4532338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4532403Z outputs = self.fnet( 2025-12-04T09:42:45.4532652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4532727Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4532974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4533060Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4533280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4533369Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4533614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4533697Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4533961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4534039Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4534318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4534449Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4534689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4534782Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4534786Z 2025-12-04T09:42:45.4534873Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4534982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4535177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4535243Z res = mod(**inputs) 2025-12-04T09:42:45.4535493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4535561Z outputs = self.fnet( 2025-12-04T09:42:45.4535803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4535888Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4536127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4536236Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4536486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4536566Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4536815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4536913Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4537163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4537245Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4537487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4537613Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4537618Z 2025-12-04T09:42:45.4537719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4537919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4537993Z res = mod(**inputs) 2025-12-04T09:42:45.4538236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4538307Z outputs = self.fnet( 2025-12-04T09:42:45.4538549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4538622Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4538874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4538958Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4539185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4539274Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4539529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4539636Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4539888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4539970Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4540229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4540332Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4540337Z 2025-12-04T09:42:45.4540449Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4540650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4540715Z res = mod(**inputs) 2025-12-04T09:42:45.4540986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4541051Z outputs = self.fnet( 2025-12-04T09:42:45.4541296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4541377Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4541623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4541716Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4541941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4542022Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4542294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4542439Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4542699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4542787Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4543049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4543165Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4543169Z 2025-12-04T09:42:45.4543278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4543510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4543587Z res = mod(**inputs) 2025-12-04T09:42:45.4543853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4543939Z outputs = self.fnet( 2025-12-04T09:42:45.4544192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4544266Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4544519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4544605Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4544836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4544915Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4545165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4545271Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4545537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4545622Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4545894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4546008Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4546012Z 2025-12-04T09:42:45.4546123Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4546323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4546388Z res = mod(**inputs) 2025-12-04T09:42:45.4546648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4546717Z outputs = self.fnet( 2025-12-04T09:42:45.4546964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4547049Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4547474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4547569Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4547794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4547875Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4548134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4548223Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4548494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4548638Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4548967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4549094Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4549345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4549431Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4549443Z 2025-12-04T09:42:45.4549551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4549751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4549853Z res = mod(**inputs) 2025-12-04T09:42:45.4550105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4550176Z outputs = self.fnet( 2025-12-04T09:42:45.4550438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4550513Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4550772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4550858Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4551085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4551176Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4551427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4551514Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4551789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4551870Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4552165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4552280Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4552530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4552651Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4552871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4553062Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4553065Z 2025-12-04T09:42:45.4553172Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4553372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4553449Z res = mod(**inputs) 2025-12-04T09:42:45.4553702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4553770Z outputs = self.fnet( 2025-12-04T09:42:45.4554027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4554101Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4554356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4554442Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4554671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4554781Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4555063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4555158Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4555421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4555500Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4555787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4555920Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4556188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4556280Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4556285Z 2025-12-04T09:42:45.4556370Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4556483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4556685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4556752Z res = mod(**inputs) 2025-12-04T09:42:45.4557009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4557078Z outputs = self.fnet( 2025-12-04T09:42:45.4557335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4557410Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4557659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4557752Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4557979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4558061Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4558319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4558418Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4558671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4558753Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4559002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4559116Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4559120Z 2025-12-04T09:42:45.4559224Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4559432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4559502Z res = mod(**inputs) 2025-12-04T09:42:45.4559747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4559821Z outputs = self.fnet( 2025-12-04T09:42:45.4560069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4560142Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4560457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4560548Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4560784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4560887Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4561173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4561283Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4561531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4561621Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4561862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4561961Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4561965Z 2025-12-04T09:42:45.4562097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4562289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4562354Z res = mod(**inputs) 2025-12-04T09:42:45.4562608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4562678Z outputs = self.fnet( 2025-12-04T09:42:45.4562927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4563009Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4563244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4563332Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4563546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4563624Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4563870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4563965Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4564218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4564295Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4564528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4564632Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4564635Z 2025-12-04T09:42:45.4564735Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4564936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4564999Z res = mod(**inputs) 2025-12-04T09:42:45.4565238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4565313Z outputs = self.fnet( 2025-12-04T09:42:45.4565552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4565622Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4565866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4565949Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4566176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4566250Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4566478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4566577Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4566826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4566910Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4567177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4567275Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4567278Z 2025-12-04T09:42:45.4567381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4567565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4567628Z res = mod(**inputs) 2025-12-04T09:42:45.4567863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4567943Z outputs = self.fnet( 2025-12-04T09:42:45.4568189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4568261Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4568492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4568580Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4568786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4568860Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4569096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4569175Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4569426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4569499Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4569763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4569882Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4570118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4570211Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4570215Z 2025-12-04T09:42:45.4570311Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4570519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4570589Z res = mod(**inputs) 2025-12-04T09:42:45.4570843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4570917Z outputs = self.fnet( 2025-12-04T09:42:45.4571152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4571222Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4571480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4571562Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4571786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4571869Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4572126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4572214Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4572463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4572555Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4572855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4572966Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4573213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4573318Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4573526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4573709Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4573730Z 2025-12-04T09:42:45.4573831Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4574024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4574091Z res = mod(**inputs) 2025-12-04T09:42:45.4574334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4574405Z outputs = self.fnet( 2025-12-04T09:42:45.4574663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4574734Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4574987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4575067Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4575289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4575366Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4575603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4575693Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4575941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4576016Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4576291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4576415Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4576661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4576743Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4576746Z 2025-12-04T09:42:45.4576824Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4576933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4577123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4577193Z res = mod(**inputs) 2025-12-04T09:42:45.4577430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4577495Z outputs = self.fnet( 2025-12-04T09:42:45.4577741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4577811Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4578048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4578141Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4578357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4578459Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4578722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4578818Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4579063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4579143Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4579378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4579485Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4579502Z 2025-12-04T09:42:45.4579602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4579803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4579868Z res = mod(**inputs) 2025-12-04T09:42:45.4580112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4580187Z outputs = self.fnet( 2025-12-04T09:42:45.4580424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4580502Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4580738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4580820Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4581041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4581119Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4581357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4581463Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4581700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4581785Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4582021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4582119Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4582122Z 2025-12-04T09:42:45.4582227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4582417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4582488Z res = mod(**inputs) 2025-12-04T09:42:45.4582727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4582793Z outputs = self.fnet( 2025-12-04T09:42:45.4583040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4583110Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4583346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4583434Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4583659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4583768Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4584036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4584135Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4584393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4584498Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4584736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4584841Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4584844Z 2025-12-04T09:42:45.4584943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4585142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4585205Z res = mod(**inputs) 2025-12-04T09:42:45.4585464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4585538Z outputs = self.fnet( 2025-12-04T09:42:45.4585773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4585855Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4586091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4586173Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4586392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4586469Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4586701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4586804Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4587036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4587121Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4587356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4587453Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4587456Z 2025-12-04T09:42:45.4587562Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4587759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4587826Z res = mod(**inputs) 2025-12-04T09:42:45.4588053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4588115Z outputs = self.fnet( 2025-12-04T09:42:45.4588353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4588423Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4588649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4588738Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4588946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4589024Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4589250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4589329Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4589576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4589651Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4589915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4590049Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4590306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4590392Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4590395Z 2025-12-04T09:42:45.4590491Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4590683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4590751Z res = mod(**inputs) 2025-12-04T09:42:45.4590987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4591075Z outputs = self.fnet( 2025-12-04T09:42:45.4591315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4591388Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4591638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4591721Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4591941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4592017Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4592264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4592350Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4592597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4592672Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4592940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4593053Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4593299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4593406Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4593615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4593796Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4593800Z 2025-12-04T09:42:45.4593900Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4594100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4594163Z res = mod(**inputs) 2025-12-04T09:42:45.4594404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4594478Z outputs = self.fnet( 2025-12-04T09:42:45.4594718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4594791Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4595035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4595117Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4595340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4595421Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4595657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4595763Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4596039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4596122Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4596389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4596513Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4596755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4596835Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4596855Z 2025-12-04T09:42:45.4596939Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4597048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4597240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4597309Z res = mod(**inputs) 2025-12-04T09:42:45.4597551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4597619Z outputs = self.fnet( 2025-12-04T09:42:45.4597876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4597951Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4598202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4598296Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4598522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4598612Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4598863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4598968Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4599228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4599310Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4599567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4599673Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4599677Z 2025-12-04T09:42:45.4599781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4599991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4600058Z res = mod(**inputs) 2025-12-04T09:42:45.4600309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4600463Z outputs = self.fnet( 2025-12-04T09:42:45.4600737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4600824Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4601098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4601191Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4601448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4601539Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4601819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4601941Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4602213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4602302Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4602546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4602649Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4602662Z 2025-12-04T09:42:45.4602763Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4602956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4603046Z res = mod(**inputs) 2025-12-04T09:42:45.4603295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4603360Z outputs = self.fnet( 2025-12-04T09:42:45.4603604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4603686Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4603921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4603999Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4604210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4604294Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4604528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4604623Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4604865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4604945Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4605188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4605286Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4605289Z 2025-12-04T09:42:45.4605388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4605584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4605647Z res = mod(**inputs) 2025-12-04T09:42:45.4605882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4605955Z outputs = self.fnet( 2025-12-04T09:42:45.4606189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4606269Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4606507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4606587Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4606806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4606883Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4607123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4607215Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4607451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4607537Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4607821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4607948Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4607960Z 2025-12-04T09:42:45.4608063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4608258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4608332Z res = mod(**inputs) 2025-12-04T09:42:45.4608575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4608642Z outputs = self.fnet( 2025-12-04T09:42:45.4608894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4609002Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4609247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4609331Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4609553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4609660Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4609901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4609984Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4610243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4610318Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4610591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4610704Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4610941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4611029Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4611032Z 2025-12-04T09:42:45.4611132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4611326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4611389Z res = mod(**inputs) 2025-12-04T09:42:45.4611623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4611695Z outputs = self.fnet( 2025-12-04T09:42:45.4611930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4612000Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4612242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4612325Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4612544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4612620Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4612853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4612943Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4613191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4613268Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4613538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4613671Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4613939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4614047Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4614253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4614436Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4614440Z 2025-12-04T09:42:45.4614539Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4614737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4614817Z res = mod(**inputs) 2025-12-04T09:42:45.4615056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4615131Z outputs = self.fnet( 2025-12-04T09:42:45.4615373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4615452Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4615691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4615773Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4615993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4616069Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4616310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4616397Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4616649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4616731Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4616998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4617121Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4617365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4617442Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4617445Z 2025-12-04T09:42:45.4617528Z cudagraph partition due to non gpu ops 2025-12-04T09:42:45.4617629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4617817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4617887Z res = mod(**inputs) 2025-12-04T09:42:45.4618126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4618189Z outputs = self.fnet( 2025-12-04T09:42:45.4618434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4618504Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4618746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4618828Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4619042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4619128Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4619367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4619486Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4619770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4619854Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4620114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4620215Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4620218Z 2025-12-04T09:42:45.4620320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4620522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4620605Z res = mod(**inputs) 2025-12-04T09:42:45.4620857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4620926Z outputs = self.fnet( 2025-12-04T09:42:45.4621186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4621266Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4621508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4621591Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4621823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4621903Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4622162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4622262Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4622513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4622604Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4622870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4622976Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4622980Z 2025-12-04T09:42:45.4623084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4623290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4623356Z res = mod(**inputs) 2025-12-04T09:42:45.4623604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4623679Z outputs = self.fnet( 2025-12-04T09:42:45.4623927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4624009Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4624259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4624344Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4624575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4624656Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4624912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4625010Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4625259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4625368Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4625955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4626062Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4626073Z 2025-12-04T09:42:45.4626177Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4626377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4626454Z res = mod(**inputs) 2025-12-04T09:42:45.4626703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4626770Z outputs = self.fnet( 2025-12-04T09:42:45.4627044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4627120Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4627379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4627467Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4627691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4627779Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4628026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:42:45.4628123Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:42:45.4628377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:42:45.4628460Z self_outputs = self.self(hidden_states) 2025-12-04T09:42:45.4628714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:42:45.4628817Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:42:45.4628820Z 2025-12-04T09:42:45.4628927Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4629137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4629202Z res = mod(**inputs) 2025-12-04T09:42:45.4629455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4629522Z outputs = self.fnet( 2025-12-04T09:42:45.4629768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4629850Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4630098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4630183Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4630415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4630495Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4630747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4630832Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4631093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4631179Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4631457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4631574Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4631848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:42:45.4631968Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4631972Z 2025-12-04T09:42:45.4632082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4632278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4632344Z res = mod(**inputs) 2025-12-04T09:42:45.4632597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4632664Z outputs = self.fnet( 2025-12-04T09:42:45.4632916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4633008Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4633257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4633354Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4633581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4633661Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4633917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4634004Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4634276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4634355Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4634641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:42:45.4634768Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:42:45.4635020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:42:45.4635141Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:45.4635362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:42:45.4635544Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:42:45.4635547Z 2025-12-04T09:42:45.4635658Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4635859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4635925Z res = mod(**inputs) 2025-12-04T09:42:45.4636185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:42:45.4636254Z outputs = self.fnet( 2025-12-04T09:42:45.4636510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:42:45.4636586Z encoder_outputs = self.encoder( 2025-12-04T09:42:45.4636833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:42:45.4636927Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:42:45.4637149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:45.4637238Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:45.4637485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:42:45.4637578Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:45.4637864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:45.4637965Z return forward_fn(*input_tensors) 2025-12-04T09:42:45.4638291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:42:45.4638437Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:42:45.4638705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:42:45.4638800Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4638804Z 2025-12-04T09:42:45.4638912Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4639124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4639217Z res = mod(**inputs) 2025-12-04T09:42:45.4639485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 680, in forward 2025-12-04T09:42:45.4639597Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:42:45.4639868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 358, in forward 2025-12-04T09:42:45.4639991Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:42:45.4640266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 339, in forward 2025-12-04T09:42:45.4640564Z hidden_states = self.transform(hidden_states) 2025-12-04T09:42:45.4640839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 320, in forward 2025-12-04T09:42:45.4640938Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:45.4640945Z 2025-12-04T09:42:45.4641056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4641273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4641348Z res = mod(**inputs) 2025-12-04T09:42:45.4641618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 680, in forward 2025-12-04T09:42:45.4641729Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:42:45.4642000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 358, in forward 2025-12-04T09:42:45.4642122Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:42:45.4642368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 340, in forward 2025-12-04T09:42:45.4642460Z hidden_states = self.decoder(hidden_states) 2025-12-04T09:42:45.4642465Z 2025-12-04T09:42:45.4642582Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:45.4642790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:45.4642862Z res = mod(**inputs) 2025-12-04T09:42:45.4643138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 685, in forward 2025-12-04T09:42:45.4643351Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:42:45.4643355Z 2025-12-04T09:42:55.1330594Z Compilation time (from dynamo_timed): 14.359251602 2025-12-04T09:42:55.1440956Z pass 2025-12-04T09:42:55.1441367Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:55.1442292Z TIMING: _recursive_pre_grad_passes:0.00574 _recursive_joint_graph_passes:0.21201 _recursive_post_grad_passes:0.06148 async_compile.wait:0.7389 code_gen:9.44101 inductor_compile:10.61029 backend_compile:12.42527 gc:0.00019 entire_frame_compile:14.35925 total_wall_time:14.35925 2025-12-04T09:42:55.1443492Z STATS: call_* op count: 232 | FakeTensorMode.__torch_dispatch__:4224 | FakeTensor.__torch_dispatch__:3268 | ProxyTorchDispatchMode.__torch_dispatch__:1259 2025-12-04T09:42:55.1444464Z Dynamo produced 1 graphs covering 232 ops with 0 graph breaks (0 unique) 2025-12-04T09:42:57.3606674Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:42:57.3607729Z import pynvml # type: ignore[import] 2025-12-04T09:43:00.7421096Z 2025-12-04T09:43:01.8750846Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:43:01.8753897Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:43:01.8772365Z cpu eval LayoutLMForMaskedLM 2025-12-04T09:43:02.5338674Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:02.7967393Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:03.0648723Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:11.2314981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2315580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2315973Z res = mod(**inputs) 2025-12-04T09:43:11.2316397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2316792Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2317273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2317765Z outputs = self.layoutlm( 2025-12-04T09:43:11.2318146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2318531Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2318992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2319455Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2319853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2320245Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2320861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2321308Z layer_outputs = layer_module( 2025-12-04T09:43:11.2321700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2322142Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2322596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2323040Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2323514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2323934Z self_outputs = self.self( 2025-12-04T09:43:11.2324330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2324809Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2325024Z 2025-12-04T09:43:11.2325135Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2325538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2325897Z res = mod(**inputs) 2025-12-04T09:43:11.2326556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2326916Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2327477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2327915Z outputs = self.layoutlm( 2025-12-04T09:43:11.2328283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2328651Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2329065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2329473Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2329877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2330236Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2330639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2331053Z layer_outputs = layer_module( 2025-12-04T09:43:11.2331425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2331824Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2332268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2332711Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2333151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2333568Z self_outputs = self.self( 2025-12-04T09:43:11.2333969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2334440Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2334640Z 2025-12-04T09:43:11.2334757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2335128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2335482Z res = mod(**inputs) 2025-12-04T09:43:11.2335835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2336354Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2336779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2337210Z outputs = self.layoutlm( 2025-12-04T09:43:11.2337570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2337969Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2338400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2338826Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2339201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2339579Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2340020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2340446Z layer_outputs = layer_module( 2025-12-04T09:43:11.2340912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2341369Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2341802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2342297Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2342799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2343234Z self_outputs = self.self( 2025-12-04T09:43:11.2343647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2344156Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2344377Z 2025-12-04T09:43:11.2344486Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2344718Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2344990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2345380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2345728Z res = mod(**inputs) 2025-12-04T09:43:11.2346077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2346456Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2346888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2347529Z outputs = self.layoutlm( 2025-12-04T09:43:11.2347892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2348274Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2348708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2349144Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2349529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2349915Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2350355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2350780Z layer_outputs = layer_module( 2025-12-04T09:43:11.2351160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2351553Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2351983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2352426Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2352867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2353361Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2353850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2354292Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2354442Z 2025-12-04T09:43:11.2354551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2354923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2355277Z res = mod(**inputs) 2025-12-04T09:43:11.2355650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2356030Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2356457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2356895Z outputs = self.layoutlm( 2025-12-04T09:43:11.2357342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2357786Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2358224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2358671Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2359057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2359449Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2359882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2360411Z layer_outputs = layer_module( 2025-12-04T09:43:11.2360832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2361236Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2361684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2362102Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2362511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2362915Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2363354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2363875Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2364362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2364805Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2364956Z 2025-12-04T09:43:11.2365065Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2365434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2365754Z res = mod(**inputs) 2025-12-04T09:43:11.2366087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2366449Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2366856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2367255Z outputs = self.layoutlm( 2025-12-04T09:43:11.2367598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2367959Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2368356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2368764Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2369133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2369480Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2369867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2370272Z layer_outputs = layer_module( 2025-12-04T09:43:11.2370632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2370992Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2371387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2371824Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2372235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2372665Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2373099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2373576Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2374018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2374447Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2374832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2375238Z return self.act(input) 2025-12-04T09:43:11.2375351Z 2025-12-04T09:43:11.2375465Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2375820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2376143Z res = mod(**inputs) 2025-12-04T09:43:11.2376467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2376810Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2377208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2377615Z outputs = self.layoutlm( 2025-12-04T09:43:11.2377962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2378318Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2378727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2379152Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2379490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2379840Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2380236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2380632Z layer_outputs = layer_module( 2025-12-04T09:43:11.2380973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2381340Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2381745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2382157Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2382556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2382949Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2383424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2383922Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2384394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2384816Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2384957Z 2025-12-04T09:43:11.2385073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2385441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2385769Z res = mod(**inputs) 2025-12-04T09:43:11.2386130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2386497Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2386950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2387360Z outputs = self.layoutlm( 2025-12-04T09:43:11.2387707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2388060Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2388469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2388881Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2389252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2389604Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2390010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2390415Z layer_outputs = layer_module( 2025-12-04T09:43:11.2390761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2391135Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2391547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2391965Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2392373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2392782Z self_outputs = self.self( 2025-12-04T09:43:11.2393175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2393656Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2393863Z 2025-12-04T09:43:11.2393971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2394338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2394683Z res = mod(**inputs) 2025-12-04T09:43:11.2395036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2395396Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2395804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2396207Z outputs = self.layoutlm( 2025-12-04T09:43:11.2396541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2396898Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2397307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2397705Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2398056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2398410Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2398811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2399207Z layer_outputs = layer_module( 2025-12-04T09:43:11.2399564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2399960Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2400518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2401006Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2401462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2401918Z self_outputs = self.self( 2025-12-04T09:43:11.2402309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2402788Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2402996Z 2025-12-04T09:43:11.2403117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2403552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2403903Z res = mod(**inputs) 2025-12-04T09:43:11.2404271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2404668Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2405112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2405543Z outputs = self.layoutlm( 2025-12-04T09:43:11.2405915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2406300Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2406733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2407192Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2407578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2407965Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2408397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2408843Z layer_outputs = layer_module( 2025-12-04T09:43:11.2409236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2409641Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2410092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2410512Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2410911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2411297Z self_outputs = self.self( 2025-12-04T09:43:11.2411678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2412152Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2412359Z 2025-12-04T09:43:11.2412450Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2412664Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2412904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2413270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2413594Z res = mod(**inputs) 2025-12-04T09:43:11.2413921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2414274Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2414677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2415114Z outputs = self.layoutlm( 2025-12-04T09:43:11.2415454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2415840Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2416231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2416634Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2416981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2417333Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2417721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2418140Z layer_outputs = layer_module( 2025-12-04T09:43:11.2418488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2418844Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2419246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2419655Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2420056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2420501Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2420952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2421360Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2421500Z 2025-12-04T09:43:11.2421612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2421962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2422285Z res = mod(**inputs) 2025-12-04T09:43:11.2422614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2422960Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2423368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2423753Z outputs = self.layoutlm( 2025-12-04T09:43:11.2424075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2424407Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2424794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2425191Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2425534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2425885Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2426298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2426704Z layer_outputs = layer_module( 2025-12-04T09:43:11.2427053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2427425Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2427839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2428265Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2428670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2429104Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2429567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2430038Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2430484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2430907Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2431047Z 2025-12-04T09:43:11.2431162Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2431526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2431855Z res = mod(**inputs) 2025-12-04T09:43:11.2432213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2432581Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2432990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2433391Z outputs = self.layoutlm( 2025-12-04T09:43:11.2433725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2434077Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2434488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2434899Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2435258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2435619Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2436032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2436452Z layer_outputs = layer_module( 2025-12-04T09:43:11.2436833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2437234Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2437675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2438124Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2438558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2438986Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2439462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2439993Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2440552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2441046Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2441468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2441845Z return self.act(input) 2025-12-04T09:43:11.2441968Z 2025-12-04T09:43:11.2442081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2442475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2442849Z res = mod(**inputs) 2025-12-04T09:43:11.2443196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2443586Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2444065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2444501Z outputs = self.layoutlm( 2025-12-04T09:43:11.2444894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2445275Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2445703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2446125Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2446495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2446872Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2447511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2447943Z layer_outputs = layer_module( 2025-12-04T09:43:11.2448321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2448729Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2449161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2449603Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2450037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2450464Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2450930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2451465Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2451932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2452351Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2452491Z 2025-12-04T09:43:11.2452597Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2452960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2453288Z res = mod(**inputs) 2025-12-04T09:43:11.2453610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2453972Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2454379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2454785Z outputs = self.layoutlm( 2025-12-04T09:43:11.2455120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2455479Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2455889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2456298Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2456645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2457001Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2457403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2457802Z layer_outputs = layer_module( 2025-12-04T09:43:11.2458160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2458540Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2459019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2459483Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2459896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2460302Z self_outputs = self.self( 2025-12-04T09:43:11.2460700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2461190Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2461406Z 2025-12-04T09:43:11.2461517Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2461914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2462235Z res = mod(**inputs) 2025-12-04T09:43:11.2462576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2462939Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2463355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2463752Z outputs = self.layoutlm( 2025-12-04T09:43:11.2464099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2464468Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2464857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2465257Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2465607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2465961Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2466357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2466759Z layer_outputs = layer_module( 2025-12-04T09:43:11.2467110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2467466Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2467874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2468286Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2468691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2469080Z self_outputs = self.self( 2025-12-04T09:43:11.2469464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2469926Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2470118Z 2025-12-04T09:43:11.2470230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2470582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2470899Z res = mod(**inputs) 2025-12-04T09:43:11.2471224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2471566Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2471964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2472363Z outputs = self.layoutlm( 2025-12-04T09:43:11.2472696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2473067Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2473534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2473939Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2474284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2474619Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2475012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2475405Z layer_outputs = layer_module( 2025-12-04T09:43:11.2475745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2476129Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2476532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2476940Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2477338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2477733Z self_outputs = self.self( 2025-12-04T09:43:11.2478111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2478583Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2478789Z 2025-12-04T09:43:11.2478872Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2479095Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2479339Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2479706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2480033Z res = mod(**inputs) 2025-12-04T09:43:11.2480440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2480807Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2481265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2481709Z outputs = self.layoutlm( 2025-12-04T09:43:11.2482093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2482470Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2482908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2483319Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2483681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2484033Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2484444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2484859Z layer_outputs = layer_module( 2025-12-04T09:43:11.2485213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2485586Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2486001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2486420Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2486833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2487320Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2487818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2488235Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2488374Z 2025-12-04T09:43:11.2488482Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2488848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2489173Z res = mod(**inputs) 2025-12-04T09:43:11.2489499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2489860Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2490298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2490703Z outputs = self.layoutlm( 2025-12-04T09:43:11.2491039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2491403Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2491808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2492206Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2492557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2492913Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2493321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2493725Z layer_outputs = layer_module( 2025-12-04T09:43:11.2494073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2494439Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2494845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2495245Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2495645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2496042Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2496473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2496947Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2497388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2497796Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2497930Z 2025-12-04T09:43:11.2498033Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2498392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2498710Z res = mod(**inputs) 2025-12-04T09:43:11.2499030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2499375Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2499776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2500181Z outputs = self.layoutlm( 2025-12-04T09:43:11.2500520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2500870Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2501288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2501681Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2502051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2502402Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2502807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2503189Z layer_outputs = layer_module( 2025-12-04T09:43:11.2503536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2503897Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2504314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2504723Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2505129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2505525Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2505954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2506420Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2506861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2507297Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2507670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2508014Z return self.act(input) 2025-12-04T09:43:11.2508138Z 2025-12-04T09:43:11.2508243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2508603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2508924Z res = mod(**inputs) 2025-12-04T09:43:11.2509261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2509626Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2510034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2510450Z outputs = self.layoutlm( 2025-12-04T09:43:11.2510796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2511163Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2511570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2511979Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2512329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2512689Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2513092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2513504Z layer_outputs = layer_module( 2025-12-04T09:43:11.2513864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2514228Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2514669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2515131Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2515588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2516050Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2516521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2517072Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2517589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2518047Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2518208Z 2025-12-04T09:43:11.2518324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2518747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2519110Z res = mod(**inputs) 2025-12-04T09:43:11.2519480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2519880Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2520389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2520841Z outputs = self.layoutlm( 2025-12-04T09:43:11.2521216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2521606Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2522092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2522505Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2522863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2523209Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2523600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2524004Z layer_outputs = layer_module( 2025-12-04T09:43:11.2524368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2524768Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2525195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2525639Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2526065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2526461Z self_outputs = self.self( 2025-12-04T09:43:11.2526854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2527363Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2527577Z 2025-12-04T09:43:11.2527697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2528081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2528436Z res = mod(**inputs) 2025-12-04T09:43:11.2528798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2529189Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2529621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2530050Z outputs = self.layoutlm( 2025-12-04T09:43:11.2530410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2530815Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2531286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2531725Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2532101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2532470Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2532900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2533331Z layer_outputs = layer_module( 2025-12-04T09:43:11.2533719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2534135Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2534572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2535015Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2535446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2535876Z self_outputs = self.self( 2025-12-04T09:43:11.2536303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2536799Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2537004Z 2025-12-04T09:43:11.2537116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2537505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2537854Z res = mod(**inputs) 2025-12-04T09:43:11.2538198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2538583Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2539013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2539437Z outputs = self.layoutlm( 2025-12-04T09:43:11.2539791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2540171Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2540598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2541022Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2541396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2541772Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2542205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2542626Z layer_outputs = layer_module( 2025-12-04T09:43:11.2543002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2543385Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2543824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2544229Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2544654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2545078Z self_outputs = self.self( 2025-12-04T09:43:11.2545482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2546061Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2546287Z 2025-12-04T09:43:11.2546375Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2546612Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2546861Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2547437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2547770Z res = mod(**inputs) 2025-12-04T09:43:11.2548100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2548469Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2548927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2549335Z outputs = self.layoutlm( 2025-12-04T09:43:11.2549677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2550047Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2550460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2550876Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2551232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2551596Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2552009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2552420Z layer_outputs = layer_module( 2025-12-04T09:43:11.2552784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2553163Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2553591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2554006Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2554420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2554888Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2555344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2555782Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2555945Z 2025-12-04T09:43:11.2556062Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2556453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2556803Z res = mod(**inputs) 2025-12-04T09:43:11.2557161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2557553Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2557991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2558416Z outputs = self.layoutlm( 2025-12-04T09:43:11.2558781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2559166Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2559586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2560024Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2560486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2560878Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2561365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2561799Z layer_outputs = layer_module( 2025-12-04T09:43:11.2562177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2562559Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2562996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2563468Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2563922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2564351Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2564839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2565379Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2565861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2566312Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2566470Z 2025-12-04T09:43:11.2566584Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2566977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2567318Z res = mod(**inputs) 2025-12-04T09:43:11.2567725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2568139Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2568557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2568946Z outputs = self.layoutlm( 2025-12-04T09:43:11.2569291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2569659Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2570061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2570454Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2570807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2571168Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2571571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2571971Z layer_outputs = layer_module( 2025-12-04T09:43:11.2572321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2572680Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2573073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2573483Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2573882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2574265Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2574721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2575255Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2575741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2576184Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2576582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2576923Z return self.act(input) 2025-12-04T09:43:11.2577034Z 2025-12-04T09:43:11.2577144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2577495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2577817Z res = mod(**inputs) 2025-12-04T09:43:11.2578187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2578536Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2578942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2579343Z outputs = self.layoutlm( 2025-12-04T09:43:11.2579688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2580051Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2580450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2580850Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2581190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2581542Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2581947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2582368Z layer_outputs = layer_module( 2025-12-04T09:43:11.2582713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2583076Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2583480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2583891Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2584285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2584676Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2585104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2585586Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2586047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2586460Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2586596Z 2025-12-04T09:43:11.2586708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2587063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2587387Z res = mod(**inputs) 2025-12-04T09:43:11.2587714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2588067Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2588458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2588858Z outputs = self.layoutlm( 2025-12-04T09:43:11.2589193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2589558Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2589983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2590381Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2590724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2591066Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2591461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2591858Z layer_outputs = layer_module( 2025-12-04T09:43:11.2592223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2592580Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2592978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2593380Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2593781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2594177Z self_outputs = self.self( 2025-12-04T09:43:11.2594565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2595029Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2595229Z 2025-12-04T09:43:11.2595334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2595696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2596019Z res = mod(**inputs) 2025-12-04T09:43:11.2596343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2596709Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2597121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2597525Z outputs = self.layoutlm( 2025-12-04T09:43:11.2597854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2598226Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2598653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2599083Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2599449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2599830Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2600262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2600763Z layer_outputs = layer_module( 2025-12-04T09:43:11.2601141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2601543Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2601979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2602416Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2602859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2603294Z self_outputs = self.self( 2025-12-04T09:43:11.2603738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2604319Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2604524Z 2025-12-04T09:43:11.2604630Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2604995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2605314Z res = mod(**inputs) 2025-12-04T09:43:11.2605646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2606006Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2606411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2606824Z outputs = self.layoutlm( 2025-12-04T09:43:11.2607168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2607531Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2607934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2608344Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2608700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2609057Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2609457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2609863Z layer_outputs = layer_module( 2025-12-04T09:43:11.2610225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2610587Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2611000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2611419Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2611833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2612231Z self_outputs = self.self( 2025-12-04T09:43:11.2612628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2613109Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2613313Z 2025-12-04T09:43:11.2613404Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2613621Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2613866Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2614232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2614549Z res = mod(**inputs) 2025-12-04T09:43:11.2614888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2615253Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2615663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2616060Z outputs = self.layoutlm( 2025-12-04T09:43:11.2616400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2616757Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2617157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2617568Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2617942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2618333Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2618732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2619134Z layer_outputs = layer_module( 2025-12-04T09:43:11.2619486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2619855Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2620258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2620695Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2621106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2621567Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2622028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2622445Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2622585Z 2025-12-04T09:43:11.2622698Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2623065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2623414Z res = mod(**inputs) 2025-12-04T09:43:11.2623772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2624126Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2624537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2624945Z outputs = self.layoutlm( 2025-12-04T09:43:11.2625289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2625637Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2626039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2626450Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2626806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2627155Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2627559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2627963Z layer_outputs = layer_module( 2025-12-04T09:43:11.2628311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2628683Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2629095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2629512Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2629916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2630315Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2630753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2631235Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2631682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2632120Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2632260Z 2025-12-04T09:43:11.2632375Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2632766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2633098Z res = mod(**inputs) 2025-12-04T09:43:11.2633437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2633801Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2634206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2634615Z outputs = self.layoutlm( 2025-12-04T09:43:11.2634978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2635329Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2635741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2636146Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2636499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2636849Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2637252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2637678Z layer_outputs = layer_module( 2025-12-04T09:43:11.2638055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2638440Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2638878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2639327Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2639757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2640183Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2640747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2641284Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2641786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2642272Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2642698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2643080Z return self.act(input) 2025-12-04T09:43:11.2643204Z 2025-12-04T09:43:11.2643317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2643718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2644070Z res = mod(**inputs) 2025-12-04T09:43:11.2644663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2645050Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2645490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2645938Z outputs = self.layoutlm( 2025-12-04T09:43:11.2646298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2646686Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2647288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2647801Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2648228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2648610Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2649043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2649476Z layer_outputs = layer_module( 2025-12-04T09:43:11.2649860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2650215Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2650631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2651021Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2651408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2651787Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2652193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2652658Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2653094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2653487Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2653618Z 2025-12-04T09:43:11.2653720Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2654070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2654378Z res = mod(**inputs) 2025-12-04T09:43:11.2654696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2655040Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2655431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2655824Z outputs = self.layoutlm( 2025-12-04T09:43:11.2656148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2656491Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2656895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2657277Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2657603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2657941Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2658326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2658709Z layer_outputs = layer_module( 2025-12-04T09:43:11.2659055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2659412Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2659809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2660209Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2660618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2661002Z self_outputs = self.self( 2025-12-04T09:43:11.2661412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2661895Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2662096Z 2025-12-04T09:43:11.2662199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2662554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2662863Z res = mod(**inputs) 2025-12-04T09:43:11.2663188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2663534Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2663923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2664317Z outputs = self.layoutlm( 2025-12-04T09:43:11.2664640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2664981Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2665366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2665748Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2666082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2666421Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2666807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2667203Z layer_outputs = layer_module( 2025-12-04T09:43:11.2667557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2667919Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2668313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2668722Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2669127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2669509Z self_outputs = self.self( 2025-12-04T09:43:11.2669889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2670384Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2670576Z 2025-12-04T09:43:11.2670687Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2671036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2671357Z res = mod(**inputs) 2025-12-04T09:43:11.2671678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2672030Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2672422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2672816Z outputs = self.layoutlm( 2025-12-04T09:43:11.2673146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2673487Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2673886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2674284Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2674626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2674997Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2675426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2675828Z layer_outputs = layer_module( 2025-12-04T09:43:11.2676176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2676573Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2677006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2677449Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2677879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2678327Z self_outputs = self.self( 2025-12-04T09:43:11.2678746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2679271Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2679494Z 2025-12-04T09:43:11.2679585Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2679826Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2680085Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2680558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2680931Z res = mod(**inputs) 2025-12-04T09:43:11.2681300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2681683Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2682073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2682479Z outputs = self.layoutlm( 2025-12-04T09:43:11.2682818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2683162Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2683563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2683963Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2684308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2684657Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2685043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2685429Z layer_outputs = layer_module( 2025-12-04T09:43:11.2685761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2686111Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2686500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2686896Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2687279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2687720Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2688151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2688557Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2688690Z 2025-12-04T09:43:11.2688792Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2689166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2689480Z res = mod(**inputs) 2025-12-04T09:43:11.2689822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2690163Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2690550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2690935Z outputs = self.layoutlm( 2025-12-04T09:43:11.2691253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2691594Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2692005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2692392Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2692727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2693068Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2693455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2693833Z layer_outputs = layer_module( 2025-12-04T09:43:11.2694174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2694528Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2694922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2695323Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2695724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2696120Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2696546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2697015Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2697447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2697855Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2697992Z 2025-12-04T09:43:11.2698111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2698459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2698773Z res = mod(**inputs) 2025-12-04T09:43:11.2699090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2699429Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2699819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2700205Z outputs = self.layoutlm( 2025-12-04T09:43:11.2700525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2700866Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2701252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2701640Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2701969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2702309Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2702696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2703102Z layer_outputs = layer_module( 2025-12-04T09:43:11.2703463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2703815Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2704207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2704606Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2705010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2705409Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2705853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2706321Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2706759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2707186Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2707555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2707879Z return self.act(input) 2025-12-04T09:43:11.2707993Z 2025-12-04T09:43:11.2708094Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2708441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2708745Z res = mod(**inputs) 2025-12-04T09:43:11.2709065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2709410Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2709802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2710186Z outputs = self.layoutlm( 2025-12-04T09:43:11.2710519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2710873Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2711267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2711673Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2712026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2712381Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2712775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2713176Z layer_outputs = layer_module( 2025-12-04T09:43:11.2713529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2713897Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2714295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2714704Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2715103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2715491Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2715920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2716408Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2716893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2717013Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2717018Z 2025-12-04T09:43:11.2717129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2717341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2717406Z res = mod(**inputs) 2025-12-04T09:43:11.2717643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2717723Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2718000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2718100Z outputs = self.layoutlm( 2025-12-04T09:43:11.2718323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2718398Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2718678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2718755Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2718992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2719072Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2719373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2719457Z layer_outputs = layer_module( 2025-12-04T09:43:11.2719697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2719787Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2720087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2720177Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2720539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2720624Z self_outputs = self.self( 2025-12-04T09:43:11.2720930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2721102Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2721106Z 2025-12-04T09:43:11.2721229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2721462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2721533Z res = mod(**inputs) 2025-12-04T09:43:11.2721767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2721857Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2722143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2722221Z outputs = self.layoutlm( 2025-12-04T09:43:11.2722459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2722538Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2722842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2722923Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2723154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2723269Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2723582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2723663Z layer_outputs = layer_module( 2025-12-04T09:43:11.2723892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2723983Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2724255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2724336Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2724599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2724695Z self_outputs = self.self( 2025-12-04T09:43:11.2724957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2725101Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2725105Z 2025-12-04T09:43:11.2725207Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2725405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2725475Z res = mod(**inputs) 2025-12-04T09:43:11.2725688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2725768Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2726030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2726102Z outputs = self.layoutlm( 2025-12-04T09:43:11.2726322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2726395Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2726664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2726747Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2726967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2727048Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2727327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2727399Z layer_outputs = layer_module( 2025-12-04T09:43:11.2727627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2727707Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2727968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2728062Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2728327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2728405Z self_outputs = self.self( 2025-12-04T09:43:11.2728667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2728808Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2728812Z 2025-12-04T09:43:11.2728903Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2728983Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2729097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2729316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2729380Z res = mod(**inputs) 2025-12-04T09:43:11.2730206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2730294Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2730558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2730637Z outputs = self.layoutlm( 2025-12-04T09:43:11.2730850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2730931Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2731218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2731297Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2731517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2731594Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2731858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2731938Z layer_outputs = layer_module( 2025-12-04T09:43:11.2732157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2732243Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2732510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2732594Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2732872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2733006Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2733285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2733370Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2733374Z 2025-12-04T09:43:11.2733481Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2733690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2733755Z res = mod(**inputs) 2025-12-04T09:43:11.2733976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2734061Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2734328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2734409Z outputs = self.layoutlm( 2025-12-04T09:43:11.2734630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2734705Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2734980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2735053Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2735276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2735351Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2735618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2735701Z layer_outputs = layer_module( 2025-12-04T09:43:11.2735926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2736026Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2736343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2736433Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2736702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2736779Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2737082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2737233Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2737501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2737591Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2737595Z 2025-12-04T09:43:11.2737703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2737902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2737974Z res = mod(**inputs) 2025-12-04T09:43:11.2738196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2738272Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2738549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2738619Z outputs = self.layoutlm( 2025-12-04T09:43:11.2738846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2738919Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2739183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2739269Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2739484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2739564Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2739832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2739902Z layer_outputs = layer_module( 2025-12-04T09:43:11.2740133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2740212Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2740477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2740570Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2740832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2740917Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2741216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2741337Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2741614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2741727Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2741952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2742044Z return self.act(input) 2025-12-04T09:43:11.2742049Z 2025-12-04T09:43:11.2742156Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2742404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2742473Z res = mod(**inputs) 2025-12-04T09:43:11.2742693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2742775Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2743044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2743125Z outputs = self.layoutlm( 2025-12-04T09:43:11.2743343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2743441Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2743723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2743799Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2744019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2744102Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2744375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2744457Z layer_outputs = layer_module( 2025-12-04T09:43:11.2744683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2744765Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2745043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2745133Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2745403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2745480Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2745782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2745926Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2746198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2746282Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2746295Z 2025-12-04T09:43:11.2746401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2746602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2746676Z res = mod(**inputs) 2025-12-04T09:43:11.2746898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2746973Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2747418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2747495Z outputs = self.layoutlm( 2025-12-04T09:43:11.2747726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2747803Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2748084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2748170Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2748397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2748520Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2748846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2748919Z layer_outputs = layer_module( 2025-12-04T09:43:11.2749150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2749231Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2749500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2749592Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2749890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2749961Z self_outputs = self.self( 2025-12-04T09:43:11.2750240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2750389Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2750393Z 2025-12-04T09:43:11.2750508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2750709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2750778Z res = mod(**inputs) 2025-12-04T09:43:11.2751010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2751081Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2751344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2751412Z outputs = self.layoutlm( 2025-12-04T09:43:11.2751622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2751704Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2751970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2752044Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2752275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2752347Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2752607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2752676Z layer_outputs = layer_module( 2025-12-04T09:43:11.2752891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2752976Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2753239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2753325Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2753585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2753656Z self_outputs = self.self( 2025-12-04T09:43:11.2753922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2754056Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2754060Z 2025-12-04T09:43:11.2754173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2754369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2754451Z res = mod(**inputs) 2025-12-04T09:43:11.2754681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2754788Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2755058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2755136Z outputs = self.layoutlm( 2025-12-04T09:43:11.2755361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2755439Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2755701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2755792Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2756016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2756091Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2756357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2756435Z layer_outputs = layer_module( 2025-12-04T09:43:11.2756663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2756749Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2757024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2757108Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2757391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2757461Z self_outputs = self.self( 2025-12-04T09:43:11.2757751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2757898Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2757901Z 2025-12-04T09:43:11.2757983Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2758073Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2758179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2758381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2758457Z res = mod(**inputs) 2025-12-04T09:43:11.2758686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2758770Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2759046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2759119Z outputs = self.layoutlm( 2025-12-04T09:43:11.2759352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2759427Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2759702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2759785Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2760007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2760089Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2760462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2760551Z layer_outputs = layer_module( 2025-12-04T09:43:11.2760832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2760920Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2761276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2761376Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2761676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2761810Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2762071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2762169Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2762172Z 2025-12-04T09:43:11.2762285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2762480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2762551Z res = mod(**inputs) 2025-12-04T09:43:11.2762772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2762843Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2763106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2763178Z outputs = self.layoutlm( 2025-12-04T09:43:11.2763386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2763466Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2763720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2763802Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2764009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2764081Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2764344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2764414Z layer_outputs = layer_module( 2025-12-04T09:43:11.2764633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2764709Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2764962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2765054Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2765304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2765380Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2765677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2765795Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2766056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2766134Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2766138Z 2025-12-04T09:43:11.2766237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2766434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2766497Z res = mod(**inputs) 2025-12-04T09:43:11.2766713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2766804Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2767124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2767201Z outputs = self.layoutlm( 2025-12-04T09:43:11.2767409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2767481Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2767752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2767823Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2768032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2768121Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2768373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2768449Z layer_outputs = layer_module( 2025-12-04T09:43:11.2768670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2768753Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2769023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2769103Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2769361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2769435Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2769726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2769850Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2770116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2770230Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2770443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2770511Z return self.act(input) 2025-12-04T09:43:11.2770514Z 2025-12-04T09:43:11.2770622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2770818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2770889Z res = mod(**inputs) 2025-12-04T09:43:11.2771107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2771178Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2771451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2771519Z outputs = self.layoutlm( 2025-12-04T09:43:11.2771733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2771811Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2772077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2772155Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2772370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2772440Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2772709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2772798Z layer_outputs = layer_module( 2025-12-04T09:43:11.2773043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2773130Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2773386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2773474Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2773723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2773796Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2774110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2774239Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2774508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2774589Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2774593Z 2025-12-04T09:43:11.2774693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2774893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2774956Z res = mod(**inputs) 2025-12-04T09:43:11.2775170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2775248Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2775510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2775587Z outputs = self.layoutlm( 2025-12-04T09:43:11.2775797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2775869Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2776136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2776207Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2776422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2776493Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2776751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2776829Z layer_outputs = layer_module( 2025-12-04T09:43:11.2777043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2777119Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2777393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2777473Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2777729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2777797Z self_outputs = self.self( 2025-12-04T09:43:11.2778053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2778201Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2778206Z 2025-12-04T09:43:11.2778305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2778505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2778585Z res = mod(**inputs) 2025-12-04T09:43:11.2778827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2778908Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2779174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2779240Z outputs = self.layoutlm( 2025-12-04T09:43:11.2779452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2779520Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2779781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2779869Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2780076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2780154Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2780406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2780473Z layer_outputs = layer_module( 2025-12-04T09:43:11.2780691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2780767Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2781023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2781100Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2781352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2781429Z self_outputs = self.self( 2025-12-04T09:43:11.2781681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2781821Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2781824Z 2025-12-04T09:43:11.2781922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2782105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2782173Z res = mod(**inputs) 2025-12-04T09:43:11.2782377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2782447Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2782712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2782781Z outputs = self.layoutlm( 2025-12-04T09:43:11.2783009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2783080Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2783329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2783407Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2783611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2783685Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2783935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2784003Z layer_outputs = layer_module( 2025-12-04T09:43:11.2784217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2784319Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2784601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2784686Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2784934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2785006Z self_outputs = self.self( 2025-12-04T09:43:11.2785253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2785388Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2785407Z 2025-12-04T09:43:11.2785494Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2785570Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2785678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2785867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2785929Z res = mod(**inputs) 2025-12-04T09:43:11.2786148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2786219Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2786479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2786553Z outputs = self.layoutlm( 2025-12-04T09:43:11.2786765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2786843Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2787105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2787178Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2787395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2787467Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2787732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2787809Z layer_outputs = layer_module( 2025-12-04T09:43:11.2788029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2788113Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2788385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2788468Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2788737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2788867Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2789134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2789214Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2789218Z 2025-12-04T09:43:11.2789318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2789521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2789584Z res = mod(**inputs) 2025-12-04T09:43:11.2789919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2790281Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2790685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2791109Z outputs = self.layoutlm( 2025-12-04T09:43:11.2791478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2791832Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2792251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2792674Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2793019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2793366Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2793769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2794193Z layer_outputs = layer_module( 2025-12-04T09:43:11.2794540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2794906Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2795306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2795725Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2796190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2796599Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2797050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2797573Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2798043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2798489Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2798646Z 2025-12-04T09:43:11.2798761Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2799148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2799490Z res = mod(**inputs) 2025-12-04T09:43:11.2799842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2800224Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2800744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2801181Z outputs = self.layoutlm( 2025-12-04T09:43:11.2801561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2801954Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2802395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2802852Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2803236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2803616Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2804039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2804470Z layer_outputs = layer_module( 2025-12-04T09:43:11.2804851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2805238Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2805675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2806145Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2806614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2807032Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2807506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2808036Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2808524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2809018Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2809442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2809815Z return self.act(input) 2025-12-04T09:43:11.2809939Z 2025-12-04T09:43:11.2810050Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2810444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2810792Z res = mod(**inputs) 2025-12-04T09:43:11.2811147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2811522Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2811958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2812391Z outputs = self.layoutlm( 2025-12-04T09:43:11.2812750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2813133Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2813567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2814003Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2814541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2814925Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2815364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2815805Z layer_outputs = layer_module( 2025-12-04T09:43:11.2816183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2816589Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2817035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2817482Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2817929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2818367Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2818859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2819409Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2820090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2820554Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2820711Z 2025-12-04T09:43:11.2820834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2821216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2821598Z res = mod(**inputs) 2025-12-04T09:43:11.2821987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2822365Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2822796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2823223Z outputs = self.layoutlm( 2025-12-04T09:43:11.2823594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2823971Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2824392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2824864Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2825241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2825622Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2826050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2826484Z layer_outputs = layer_module( 2025-12-04T09:43:11.2826860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2827261Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2827689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2828134Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2828573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2828999Z self_outputs = self.self( 2025-12-04T09:43:11.2829425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2829936Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2830153Z 2025-12-04T09:43:11.2830271Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2830657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2831007Z res = mod(**inputs) 2025-12-04T09:43:11.2831355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2831735Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2832163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2832593Z outputs = self.layoutlm( 2025-12-04T09:43:11.2832954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2833326Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2833759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2834188Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2834562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2834929Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2835360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2835772Z layer_outputs = layer_module( 2025-12-04T09:43:11.2836114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2836495Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2836938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2837360Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2837769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2838175Z self_outputs = self.self( 2025-12-04T09:43:11.2838561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2839019Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2839230Z 2025-12-04T09:43:11.2839334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2839694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2840018Z res = mod(**inputs) 2025-12-04T09:43:11.2840418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2840781Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2841218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2841676Z outputs = self.layoutlm( 2025-12-04T09:43:11.2842033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2842419Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2842853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2843269Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2843616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2843970Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2844366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2844752Z layer_outputs = layer_module( 2025-12-04T09:43:11.2845104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2845463Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2845865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2846266Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2846669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2847224Z self_outputs = self.self( 2025-12-04T09:43:11.2847609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2848082Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2848291Z 2025-12-04T09:43:11.2848375Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2848591Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2848821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2849179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2849502Z res = mod(**inputs) 2025-12-04T09:43:11.2849841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2850203Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2850588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2851033Z outputs = self.layoutlm( 2025-12-04T09:43:11.2851409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2851761Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2852163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2852555Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2852908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2853261Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2853658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2854082Z layer_outputs = layer_module( 2025-12-04T09:43:11.2854445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2854799Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2855187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2855577Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2855967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2856405Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2856833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2857234Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2857375Z 2025-12-04T09:43:11.2857476Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2857829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2858138Z res = mod(**inputs) 2025-12-04T09:43:11.2858453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2858796Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2859187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2859567Z outputs = self.layoutlm( 2025-12-04T09:43:11.2859892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2860230Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2860610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2860998Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2861339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2861679Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2862056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2862439Z layer_outputs = layer_module( 2025-12-04T09:43:11.2862786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2863132Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2863523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2863936Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2864335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2864750Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2865213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2865691Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2866144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2866538Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2866679Z 2025-12-04T09:43:11.2866780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2867134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2867462Z res = mod(**inputs) 2025-12-04T09:43:11.2867790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2868136Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2868533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2868914Z outputs = self.layoutlm( 2025-12-04T09:43:11.2869247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2869592Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2869977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2870375Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2870719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2871061Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2871448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2871845Z layer_outputs = layer_module( 2025-12-04T09:43:11.2872195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2872550Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2872945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2873352Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2873746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2874129Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2874559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2875040Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2875481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2875914Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2876293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2876632Z return self.act(input) 2025-12-04T09:43:11.2876743Z 2025-12-04T09:43:11.2876854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2877204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2877525Z res = mod(**inputs) 2025-12-04T09:43:11.2877846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2878207Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2878647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2879042Z outputs = self.layoutlm( 2025-12-04T09:43:11.2879377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2879715Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2880113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2880603Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2880956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2881350Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2881756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2882167Z layer_outputs = layer_module( 2025-12-04T09:43:11.2882525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2882905Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2883326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2883754Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2884166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2884549Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2884971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2885437Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2885897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2886292Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2886425Z 2025-12-04T09:43:11.2886533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2886867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2887177Z res = mod(**inputs) 2025-12-04T09:43:11.2887492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2887840Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2888232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2888622Z outputs = self.layoutlm( 2025-12-04T09:43:11.2888954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2889293Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2889689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2890079Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2890424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2890764Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2891158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2891557Z layer_outputs = layer_module( 2025-12-04T09:43:11.2891894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2892279Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2892709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2893114Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2893502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2893891Z self_outputs = self.self( 2025-12-04T09:43:11.2894268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2894729Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2894939Z 2025-12-04T09:43:11.2895048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2895400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2895713Z res = mod(**inputs) 2025-12-04T09:43:11.2896021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2896360Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2896744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2897128Z outputs = self.layoutlm( 2025-12-04T09:43:11.2897447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2897788Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2898169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2898557Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2898893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2899237Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2899623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2900006Z layer_outputs = layer_module( 2025-12-04T09:43:11.2900346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2900698Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2901087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2901477Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2901869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2902254Z self_outputs = self.self( 2025-12-04T09:43:11.2902619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2903065Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2903257Z 2025-12-04T09:43:11.2903355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2903702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2904004Z res = mod(**inputs) 2025-12-04T09:43:11.2904318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2904658Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2905048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2905455Z outputs = self.layoutlm( 2025-12-04T09:43:11.2905781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2906164Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2906552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2906960Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2907313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2907674Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2908073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2908488Z layer_outputs = layer_module( 2025-12-04T09:43:11.2908837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2909194Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2909600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2910005Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2910412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2910794Z self_outputs = self.self( 2025-12-04T09:43:11.2911182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2911650Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2911850Z 2025-12-04T09:43:11.2911937Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2912147Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2912383Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2912741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2913059Z res = mod(**inputs) 2025-12-04T09:43:11.2913386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2913740Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2914136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2914528Z outputs = self.layoutlm( 2025-12-04T09:43:11.2914863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2915212Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2915601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2915999Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2916346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2916696Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2917115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2917546Z layer_outputs = layer_module( 2025-12-04T09:43:11.2917931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2918316Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2918753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2919197Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2919631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2920197Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2920966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2921429Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2921589Z 2025-12-04T09:43:11.2921702Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2922053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2922381Z res = mod(**inputs) 2025-12-04T09:43:11.2922705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2923077Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2923473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2923868Z outputs = self.layoutlm( 2025-12-04T09:43:11.2924207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2924555Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2924958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2925355Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2925699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2926044Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2926455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2926861Z layer_outputs = layer_module( 2025-12-04T09:43:11.2927210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2927585Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2927995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2928420Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2928825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2929229Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2929663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2930152Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2930597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2931036Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2931181Z 2025-12-04T09:43:11.2931303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2931680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2932026Z res = mod(**inputs) 2025-12-04T09:43:11.2932373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2932758Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2933161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2933569Z outputs = self.layoutlm( 2025-12-04T09:43:11.2933915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2934290Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2934737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2935148Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2935502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2935854Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2936267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2936700Z layer_outputs = layer_module( 2025-12-04T09:43:11.2937072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2937456Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2937866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2938292Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2938699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2939106Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2939555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2940039Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2940493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2940952Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2941351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2941707Z return self.act(input) 2025-12-04T09:43:11.2941824Z 2025-12-04T09:43:11.2941933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2942302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2942632Z res = mod(**inputs) 2025-12-04T09:43:11.2942958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2943321Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2943734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2944147Z outputs = self.layoutlm( 2025-12-04T09:43:11.2944498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2944883Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2945312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2945723Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2946086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2946450Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2946861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2947412Z layer_outputs = layer_module( 2025-12-04T09:43:11.2947801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2948214Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2948648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2949153Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2949649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2950079Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2950538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2951068Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2951562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2952008Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2952190Z 2025-12-04T09:43:11.2952302Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2952694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2953055Z res = mod(**inputs) 2025-12-04T09:43:11.2953405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2953788Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2954223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2954657Z outputs = self.layoutlm( 2025-12-04T09:43:11.2955018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2955402Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2955832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2956262Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2956631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2957001Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2957408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2957826Z layer_outputs = layer_module( 2025-12-04T09:43:11.2958204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2958599Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2959039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2959479Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2959920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2960408Z self_outputs = self.self( 2025-12-04T09:43:11.2960843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:43:11.2961361Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2961590Z 2025-12-04T09:43:11.2961708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2962115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2962457Z res = mod(**inputs) 2025-12-04T09:43:11.2962816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2963202Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2963638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2964098Z outputs = self.layoutlm( 2025-12-04T09:43:11.2964447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2964861Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2965300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2965715Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2966074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2966440Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2966845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2967284Z layer_outputs = layer_module( 2025-12-04T09:43:11.2967635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2968004Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2968403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2968811Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2969223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2969629Z self_outputs = self.self( 2025-12-04T09:43:11.2970010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:43:11.2970474Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2970664Z 2025-12-04T09:43:11.2970777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2971136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2971474Z res = mod(**inputs) 2025-12-04T09:43:11.2971800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2972149Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2972557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2972959Z outputs = self.layoutlm( 2025-12-04T09:43:11.2973301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2973658Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2974063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2974468Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2974823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2975175Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2975583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2975986Z layer_outputs = layer_module( 2025-12-04T09:43:11.2976333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2976702Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2977111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2977525Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2977942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:43:11.2978381Z self_outputs = self.self( 2025-12-04T09:43:11.2978808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:43:11.2979289Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:43:11.2979490Z 2025-12-04T09:43:11.2979575Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2979795Z cudagraph partition due to non gpu ops 2025-12-04T09:43:11.2980036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2980394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2980721Z res = mod(**inputs) 2025-12-04T09:43:11.2981058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2981453Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2981858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2982267Z outputs = self.layoutlm( 2025-12-04T09:43:11.2982616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2982969Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2983378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2983785Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2984139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2984487Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2984895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2984969Z layer_outputs = layer_module( 2025-12-04T09:43:11.2985206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2985290Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2985576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:43:11.2985660Z self_attention_outputs = self.attention( 2025-12-04T09:43:11.2985924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:43:11.2986060Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:43:11.2986325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:43:11.2986412Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2986422Z 2025-12-04T09:43:11.2986530Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2986734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2986814Z res = mod(**inputs) 2025-12-04T09:43:11.2987037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2987113Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2987390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2987464Z outputs = self.layoutlm( 2025-12-04T09:43:11.2987690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2987768Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2988036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2988158Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2988421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2988497Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2988767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2988838Z layer_outputs = layer_module( 2025-12-04T09:43:11.2989070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2989149Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2989420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2989528Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2989779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2989862Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2990154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2990272Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2990536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:43:11.2990616Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2990619Z 2025-12-04T09:43:11.2990719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2990920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2990983Z res = mod(**inputs) 2025-12-04T09:43:11.2991199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2991272Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2991539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2991616Z outputs = self.layoutlm( 2025-12-04T09:43:11.2991828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2991905Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2992184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2992255Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2992484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2992556Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2992818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2992898Z layer_outputs = layer_module( 2025-12-04T09:43:11.2993128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2993210Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2993464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2993545Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2993798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2993875Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2994162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:43:11.2994307Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:11.2994595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:43:11.2994712Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:11.2994919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:43:11.2994988Z return self.act(input) 2025-12-04T09:43:11.2994992Z 2025-12-04T09:43:11.2995098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2995289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2995378Z res = mod(**inputs) 2025-12-04T09:43:11.2995588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2995660Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2995928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:43:11.2995998Z outputs = self.layoutlm( 2025-12-04T09:43:11.2996205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2996283Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2996541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:43:11.2996619Z encoder_outputs = self.encoder( 2025-12-04T09:43:11.2996833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2996906Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.2997176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:43:11.2997249Z layer_outputs = layer_module( 2025-12-04T09:43:11.2997476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:11.2997557Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:11.2997822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:43:11.2997913Z layer_output = apply_chunking_to_forward( 2025-12-04T09:43:11.2998181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:43:11.2998259Z return forward_fn(*input_tensors) 2025-12-04T09:43:11.2998568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:43:11.2998704Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:43:11.2998987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:43:11.2999072Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.2999075Z 2025-12-04T09:43:11.2999178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.2999390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.2999456Z res = mod(**inputs) 2025-12-04T09:43:11.2999687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.2999762Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.3000034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 735, in forward 2025-12-04T09:43:11.3000156Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:43:11.3000536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 456, in forward 2025-12-04T09:43:11.3000659Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:43:11.3000942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 444, in forward 2025-12-04T09:43:11.3001051Z hidden_states = self.transform(hidden_states) 2025-12-04T09:43:11.3001369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 419, in forward 2025-12-04T09:43:11.3001462Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:11.3001487Z 2025-12-04T09:43:11.3001613Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.3001825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.3001893Z res = mod(**inputs) 2025-12-04T09:43:11.3002130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.3002208Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.3002481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 735, in forward 2025-12-04T09:43:11.3002582Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:43:11.3002859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 456, in forward 2025-12-04T09:43:11.3002973Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:43:11.3003259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 445, in forward 2025-12-04T09:43:11.3003353Z hidden_states = self.decoder(hidden_states) 2025-12-04T09:43:11.3003358Z 2025-12-04T09:43:11.3003469Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:11.3003672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:11.3003738Z res = mod(**inputs) 2025-12-04T09:43:11.3003971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:43:11.3004047Z output = func(self, *args, **kwargs) 2025-12-04T09:43:11.3004326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 740, in forward 2025-12-04T09:43:11.3004400Z masked_lm_loss = loss_fct( 2025-12-04T09:43:11.3004403Z 2025-12-04T09:43:21.4726807Z Compilation time (from dynamo_timed): 17.54359991 2025-12-04T09:43:21.4805160Z pass 2025-12-04T09:43:21.4805495Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:21.4806399Z TIMING: _recursive_pre_grad_passes:0.00794 _recursive_joint_graph_passes:0.47076 _recursive_post_grad_passes:0.07009 async_compile.wait:0.68557 code_gen:9.50026 inductor_compile:10.84898 backend_compile:14.45444 gc:0.00025 entire_frame_compile:17.5436 total_wall_time:17.5436 2025-12-04T09:43:21.4807360Z STATS: call_* op count: 432 | FakeTensorMode.__torch_dispatch__:8841 | FakeTensor.__torch_dispatch__:4457 | ProxyTorchDispatchMode.__torch_dispatch__:2621 2025-12-04T09:43:21.4807857Z Dynamo produced 1 graphs covering 432 ops with 0 graph breaks (0 unique) 2025-12-04T09:43:23.8856264Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:43:23.8858135Z import pynvml # type: ignore[import] 2025-12-04T09:43:27.1730094Z 2025-12-04T09:43:32.8322524Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:43:32.8322823Z loading model: 0it [00:05, ?it/s] 2025-12-04T09:43:32.8349674Z cpu eval M2M100ForConditionalGeneration 2025-12-04T09:43:33.6326622Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:33.9681462Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:34.3876869Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:50.0144158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0146214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0146584Z res = mod(**inputs) 2025-12-04T09:43:50.0147517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0147939Z outputs = self.model( 2025-12-04T09:43:50.0155850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0156504Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0157620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 849, in forward 2025-12-04T09:43:50.0158404Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-12-04T09:43:50.0158865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:43:50.0159281Z return func(*args, **kwargs) 2025-12-04T09:43:50.0159774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:43:50.0160669Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:43:50.0161417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-12-04T09:43:50.0161951Z mask = input_ids.ne(padding_idx).int() 2025-12-04T09:43:50.0162121Z 2025-12-04T09:43:50.0162285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0162725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0163098Z res = mod(**inputs) 2025-12-04T09:43:50.0163504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0164005Z outputs = self.model( 2025-12-04T09:43:50.0164412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0164843Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0165261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1098, in forward 2025-12-04T09:43:50.0165787Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-12-04T09:43:50.0166289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:43:50.0166684Z return func(*args, **kwargs) 2025-12-04T09:43:50.0167086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:43:50.0167649Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:43:50.0168276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-12-04T09:43:50.0168768Z mask = input_ids.ne(padding_idx).int() 2025-12-04T09:43:50.0169100Z 2025-12-04T09:43:50.0169189Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0169422Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0169732Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0169948Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0170168Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0170385Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0170601Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0170809Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0171021Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0171230Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0171430Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0171638Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0171918Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0172289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0172629Z res = mod(**inputs) 2025-12-04T09:43:50.0173015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0173404Z outputs = self.model( 2025-12-04T09:43:50.0173786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0174182Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0174581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 849, in forward 2025-12-04T09:43:50.0175029Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-12-04T09:43:50.0175445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:43:50.0175819Z return func(*args, **kwargs) 2025-12-04T09:43:50.0176201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:43:50.0176722Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:43:50.0177306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T09:43:50.0177869Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:43:50.0178111Z 2025-12-04T09:43:50.0178226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0178590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0178923Z res = mod(**inputs) 2025-12-04T09:43:50.0179299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0179688Z outputs = self.model( 2025-12-04T09:43:50.0180068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0180462Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0180852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 849, in forward 2025-12-04T09:43:50.0181280Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-12-04T09:43:50.0181733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:43:50.0182102Z return func(*args, **kwargs) 2025-12-04T09:43:50.0182479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:43:50.0197498Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:43:50.0198266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T09:43:50.0199003Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:43:50.0199291Z 2025-12-04T09:43:50.0199430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0199857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0200236Z res = mod(**inputs) 2025-12-04T09:43:50.0200824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0201268Z outputs = self.model( 2025-12-04T09:43:50.0201754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0202178Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0202606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0203029Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0203419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0203811Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0204238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0204692Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0205118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0205530Z return func(*args, **kwargs) 2025-12-04T09:43:50.0205936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0206435Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0206657Z 2025-12-04T09:43:50.0206776Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0207178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0207528Z res = mod(**inputs) 2025-12-04T09:43:50.0207922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0208330Z outputs = self.model( 2025-12-04T09:43:50.0208731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0209153Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0209559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0209976Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0210364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0210767Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0211179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0211617Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0212039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0212419Z return func(*args, **kwargs) 2025-12-04T09:43:50.0212798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0213199Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0213364Z 2025-12-04T09:43:50.0213480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0213879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0214222Z res = mod(**inputs) 2025-12-04T09:43:50.0214598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0215017Z outputs = self.model( 2025-12-04T09:43:50.0215406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0215830Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0216245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0216684Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0217057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0217453Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0217878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0218306Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0218717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0219099Z return func(*args, **kwargs) 2025-12-04T09:43:50.0219478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0219890Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0220049Z 2025-12-04T09:43:50.0220138Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0220399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0220782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0221128Z res = mod(**inputs) 2025-12-04T09:43:50.0221522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0221936Z outputs = self.model( 2025-12-04T09:43:50.0222323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0222749Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0223162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0223574Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0223944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0224336Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0224761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0225188Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0225610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0226013Z return func(*args, **kwargs) 2025-12-04T09:43:50.0226416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0226853Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0227337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0227862Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0228081Z 2025-12-04T09:43:50.0228211Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0228633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0228989Z res = mod(**inputs) 2025-12-04T09:43:50.0229385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0229794Z outputs = self.model( 2025-12-04T09:43:50.0230190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0230604Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0231012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0231437Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0231825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0232229Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0232752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0233209Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0233647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0234123Z return func(*args, **kwargs) 2025-12-04T09:43:50.0234539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0234969Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0235131Z 2025-12-04T09:43:50.0235247Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0235636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0235989Z res = mod(**inputs) 2025-12-04T09:43:50.0236469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0236895Z outputs = self.model( 2025-12-04T09:43:50.0237288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0237706Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0238128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0238560Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0238930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0239337Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0239772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0240268Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0240530Z 2025-12-04T09:43:50.0240651Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0241062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0241425Z res = mod(**inputs) 2025-12-04T09:43:50.0241838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0242262Z outputs = self.model( 2025-12-04T09:43:50.0242665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0243097Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0243519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0243975Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0244397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0244801Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0245229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0245713Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0245904Z 2025-12-04T09:43:50.0246030Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0246420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0246799Z res = mod(**inputs) 2025-12-04T09:43:50.0247476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0247918Z outputs = self.model( 2025-12-04T09:43:50.0248328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0248769Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0249252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0249694Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0250084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0250495Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0250935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0251371Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0251534Z 2025-12-04T09:43:50.0251649Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0252087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0252444Z res = mod(**inputs) 2025-12-04T09:43:50.0252839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0253260Z outputs = self.model( 2025-12-04T09:43:50.0253664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0254091Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0254535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0254964Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0255354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0255759Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0256210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0257450Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0257878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0258280Z return func(*args, **kwargs) 2025-12-04T09:43:50.0258711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0259235Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0259459Z 2025-12-04T09:43:50.0259580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0260042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0260390Z res = mod(**inputs) 2025-12-04T09:43:50.0260851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0261260Z outputs = self.model( 2025-12-04T09:43:50.0261650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0262073Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0262481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0262904Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0263316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0263711Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0264127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0264581Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0265005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0265410Z return func(*args, **kwargs) 2025-12-04T09:43:50.0265807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0266230Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0266375Z 2025-12-04T09:43:50.0266501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0266892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0267233Z res = mod(**inputs) 2025-12-04T09:43:50.0267623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0268034Z outputs = self.model( 2025-12-04T09:43:50.0268422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0268838Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0269248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0269661Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0270032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0270425Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0270844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0271266Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0271690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0272093Z return func(*args, **kwargs) 2025-12-04T09:43:50.0272494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0272912Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0273070Z 2025-12-04T09:43:50.0273157Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0273418Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0273797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0274140Z res = mod(**inputs) 2025-12-04T09:43:50.0274539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0274949Z outputs = self.model( 2025-12-04T09:43:50.0275363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0275789Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0276205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0276623Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0276996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0277391Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0277810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0278255Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0278685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0279096Z return func(*args, **kwargs) 2025-12-04T09:43:50.0279510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0279953Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0280563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0281107Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0281311Z 2025-12-04T09:43:50.0281436Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0281832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0282188Z res = mod(**inputs) 2025-12-04T09:43:50.0282565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0282948Z outputs = self.model( 2025-12-04T09:43:50.0283326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0283720Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0284108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0284494Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0284851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0285219Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0285606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0286016Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0286417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0286795Z return func(*args, **kwargs) 2025-12-04T09:43:50.0287169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0287573Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0287713Z 2025-12-04T09:43:50.0287827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0288193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0288509Z res = mod(**inputs) 2025-12-04T09:43:50.0288876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0289262Z outputs = self.model( 2025-12-04T09:43:50.0289650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0290079Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0290457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0290842Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0291192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0291542Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0291914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0292346Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0292526Z 2025-12-04T09:43:50.0292632Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0292991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0293299Z res = mod(**inputs) 2025-12-04T09:43:50.0293640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0294005Z outputs = self.model( 2025-12-04T09:43:50.0294358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0294727Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0295086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0295453Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0295790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0296132Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0296507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0296921Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0297087Z 2025-12-04T09:43:50.0297199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0297545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0297862Z res = mod(**inputs) 2025-12-04T09:43:50.0298220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0298594Z outputs = self.model( 2025-12-04T09:43:50.0298951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0299335Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0299710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0300083Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0300428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0300817Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0301203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0301587Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0301732Z 2025-12-04T09:43:50.0301836Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0302189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0302495Z res = mod(**inputs) 2025-12-04T09:43:50.0302871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0303245Z outputs = self.model( 2025-12-04T09:43:50.0303642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0304029Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0304405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0304785Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0305134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0305489Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0305895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:43:50.0306291Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.0306425Z 2025-12-04T09:43:50.0306530Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0306887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0307213Z res = mod(**inputs) 2025-12-04T09:43:50.0307574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0307944Z outputs = self.model( 2025-12-04T09:43:50.0308307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0308696Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0309070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0309451Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0309801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0310175Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0310566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0310977Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0311379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0311762Z return func(*args, **kwargs) 2025-12-04T09:43:50.0312140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0312611Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0312815Z 2025-12-04T09:43:50.0312930Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0313282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0313609Z res = mod(**inputs) 2025-12-04T09:43:50.0313982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0314378Z outputs = self.model( 2025-12-04T09:43:50.0314737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0315120Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0315524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0315920Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0316277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0316676Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0317101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0317520Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0317917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0318305Z return func(*args, **kwargs) 2025-12-04T09:43:50.0318689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0319106Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0319252Z 2025-12-04T09:43:50.0319384Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0319771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0320115Z res = mod(**inputs) 2025-12-04T09:43:50.0320594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0321031Z outputs = self.model( 2025-12-04T09:43:50.0321450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0321885Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0322324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0322750Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0323122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0323508Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0323884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0324330Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0324771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0325185Z return func(*args, **kwargs) 2025-12-04T09:43:50.0325614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0326055Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0326211Z 2025-12-04T09:43:50.0326311Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0326570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0326975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0327337Z res = mod(**inputs) 2025-12-04T09:43:50.0327753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0328172Z outputs = self.model( 2025-12-04T09:43:50.0328600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0329033Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0329447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0329873Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0330263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0330684Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0331108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0331582Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0332006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0332440Z return func(*args, **kwargs) 2025-12-04T09:43:50.0332844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0333286Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0333767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0334272Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0334486Z 2025-12-04T09:43:50.0334594Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0334979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0335307Z res = mod(**inputs) 2025-12-04T09:43:50.0335667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0336063Z outputs = self.model( 2025-12-04T09:43:50.0336438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0336827Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0337224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0337618Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0337973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0338338Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0338738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0339176Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0339605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0340003Z return func(*args, **kwargs) 2025-12-04T09:43:50.0340402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0340826Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0340971Z 2025-12-04T09:43:50.0341082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0341462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0341808Z res = mod(**inputs) 2025-12-04T09:43:50.0342202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0342606Z outputs = self.model( 2025-12-04T09:43:50.0343011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0343403Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0343782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0344168Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0344526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0344896Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0345288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0345730Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0345934Z 2025-12-04T09:43:50.0346041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0346449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0346769Z res = mod(**inputs) 2025-12-04T09:43:50.0347339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0347749Z outputs = self.model( 2025-12-04T09:43:50.0348124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0348528Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0348923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0349360Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0349711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0350083Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0350484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0350929Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0351105Z 2025-12-04T09:43:50.0351212Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0351587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0351916Z res = mod(**inputs) 2025-12-04T09:43:50.0352280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0352697Z outputs = self.model( 2025-12-04T09:43:50.0353092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0353511Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0353935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0354395Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0354750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0355129Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0355557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0355979Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0356128Z 2025-12-04T09:43:50.0356247Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0356627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0356987Z res = mod(**inputs) 2025-12-04T09:43:50.0357388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0357813Z outputs = self.model( 2025-12-04T09:43:50.0358209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0358627Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0359042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0359444Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0359818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0360214Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0360692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0361181Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0361668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0362101Z return func(*args, **kwargs) 2025-12-04T09:43:50.0362525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0363070Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0363305Z 2025-12-04T09:43:50.0363419Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0363812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0364183Z res = mod(**inputs) 2025-12-04T09:43:50.0364600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0365023Z outputs = self.model( 2025-12-04T09:43:50.0365426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0365841Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0366233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0366623Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0366974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0367352Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0367751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0368165Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0368568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0368956Z return func(*args, **kwargs) 2025-12-04T09:43:50.0369341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0369741Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0369882Z 2025-12-04T09:43:50.0369990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0370356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0370681Z res = mod(**inputs) 2025-12-04T09:43:50.0371042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0371432Z outputs = self.model( 2025-12-04T09:43:50.0371808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0372207Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0372598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0372991Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0373349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0373715Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0374110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0374521Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0374925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0375324Z return func(*args, **kwargs) 2025-12-04T09:43:50.0375709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0376148Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0376296Z 2025-12-04T09:43:50.0376387Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0376627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0376992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0377317Z res = mod(**inputs) 2025-12-04T09:43:50.0377678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0378067Z outputs = self.model( 2025-12-04T09:43:50.0378458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0378855Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0379235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0379620Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0379977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0380336Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0380726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0381132Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0381526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0381903Z return func(*args, **kwargs) 2025-12-04T09:43:50.0382266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0382670Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0383113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0383582Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0383768Z 2025-12-04T09:43:50.0383868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0384218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0384521Z res = mod(**inputs) 2025-12-04T09:43:50.0384882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0385271Z outputs = self.model( 2025-12-04T09:43:50.0385645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0386031Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0386420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0386810Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0387150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0387513Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0387903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0388308Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0388707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0389121Z return func(*args, **kwargs) 2025-12-04T09:43:50.0389503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0389936Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0390077Z 2025-12-04T09:43:50.0390195Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0390558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0390883Z res = mod(**inputs) 2025-12-04T09:43:50.0391243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0391629Z outputs = self.model( 2025-12-04T09:43:50.0391997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0392397Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0392770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0393149Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0393500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0393865Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0394263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0394699Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0394874Z 2025-12-04T09:43:50.0394987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0395345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0395677Z res = mod(**inputs) 2025-12-04T09:43:50.0396048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0396443Z outputs = self.model( 2025-12-04T09:43:50.0396812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0397203Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0397611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0398015Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0398397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0398787Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0399206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0399669Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0399862Z 2025-12-04T09:43:50.0399974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0400439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0400798Z res = mod(**inputs) 2025-12-04T09:43:50.0401183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0401598Z outputs = self.model( 2025-12-04T09:43:50.0401995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0402408Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0402824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0403235Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0403619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0404020Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0404425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0404829Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0404973Z 2025-12-04T09:43:50.0405077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0405444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0405769Z res = mod(**inputs) 2025-12-04T09:43:50.0406141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0406541Z outputs = self.model( 2025-12-04T09:43:50.0406916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0407312Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0407705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0408086Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0408441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0408807Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0409193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:43:50.0409593Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.0409740Z 2025-12-04T09:43:50.0409848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0410213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0410533Z res = mod(**inputs) 2025-12-04T09:43:50.0410906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0411296Z outputs = self.model( 2025-12-04T09:43:50.0411661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0412052Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0412436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0412829Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0413177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0413549Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0413946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0414356Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0414747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0415122Z return func(*args, **kwargs) 2025-12-04T09:43:50.0415504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0415967Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0416193Z 2025-12-04T09:43:50.0416305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0416691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0417036Z res = mod(**inputs) 2025-12-04T09:43:50.0417451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0417876Z outputs = self.model( 2025-12-04T09:43:50.0418323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0418740Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0419159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0419571Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0419945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0420334Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0420789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0421230Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0421670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0422083Z return func(*args, **kwargs) 2025-12-04T09:43:50.0422506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0422938Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0423085Z 2025-12-04T09:43:50.0423199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0423596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0423948Z res = mod(**inputs) 2025-12-04T09:43:50.0424353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0424762Z outputs = self.model( 2025-12-04T09:43:50.0425173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0425603Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0426028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0426437Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0426821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0427225Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0427636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0428074Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0428504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0428916Z return func(*args, **kwargs) 2025-12-04T09:43:50.0429327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0429741Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0429886Z 2025-12-04T09:43:50.0429978Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0430223Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0430588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0430919Z res = mod(**inputs) 2025-12-04T09:43:50.0431287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0431676Z outputs = self.model( 2025-12-04T09:43:50.0432049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0432467Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0432963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0433355Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0433711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0434077Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0434462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0434874Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0435296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0435685Z return func(*args, **kwargs) 2025-12-04T09:43:50.0436074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0436500Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0436952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0437433Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0437626Z 2025-12-04T09:43:50.0437732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0438095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0438424Z res = mod(**inputs) 2025-12-04T09:43:50.0438787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0439181Z outputs = self.model( 2025-12-04T09:43:50.0439557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0439955Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0440336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0440828Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0441223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0441622Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0442056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0442513Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0442940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0443338Z return func(*args, **kwargs) 2025-12-04T09:43:50.0443749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0444174Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0444322Z 2025-12-04T09:43:50.0444442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0444819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0445163Z res = mod(**inputs) 2025-12-04T09:43:50.0445554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0445963Z outputs = self.model( 2025-12-04T09:43:50.0446364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0446805Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0447406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0447882Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0448262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0448656Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0449064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0449544Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0449736Z 2025-12-04T09:43:50.0449851Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0450268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0450607Z res = mod(**inputs) 2025-12-04T09:43:50.0451002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0451418Z outputs = self.model( 2025-12-04T09:43:50.0451810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0452216Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0452622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0453033Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0453399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0453790Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0454206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0454670Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0454852Z 2025-12-04T09:43:50.0454966Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0455347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0455693Z res = mod(**inputs) 2025-12-04T09:43:50.0456072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0456493Z outputs = self.model( 2025-12-04T09:43:50.0456885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0457307Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0457710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0458105Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0458464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0458843Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0459233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0459633Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0459773Z 2025-12-04T09:43:50.0459888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0460244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0460574Z res = mod(**inputs) 2025-12-04T09:43:50.0460945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0461337Z outputs = self.model( 2025-12-04T09:43:50.0461730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0462158Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0462548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0462935Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0463292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0463675Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0464079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0464511Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0464932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0465327Z return func(*args, **kwargs) 2025-12-04T09:43:50.0465720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0466195Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0466413Z 2025-12-04T09:43:50.0466523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0466897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0467227Z res = mod(**inputs) 2025-12-04T09:43:50.0467613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0468064Z outputs = self.model( 2025-12-04T09:43:50.0468474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0468896Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0469326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0469725Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0470088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0470454Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0470857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0471276Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0471678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0472073Z return func(*args, **kwargs) 2025-12-04T09:43:50.0472464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0472874Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0473019Z 2025-12-04T09:43:50.0473130Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0473506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0473838Z res = mod(**inputs) 2025-12-04T09:43:50.0474206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0474618Z outputs = self.model( 2025-12-04T09:43:50.0475015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0475439Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0475849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0476290Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0476713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0477105Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0477516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0477953Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0478376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0478772Z return func(*args, **kwargs) 2025-12-04T09:43:50.0479196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0479625Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0479779Z 2025-12-04T09:43:50.0479872Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0480127Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0480610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0480983Z res = mod(**inputs) 2025-12-04T09:43:50.0481385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0481856Z outputs = self.model( 2025-12-04T09:43:50.0482257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0482684Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0483095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0483519Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0483902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0484303Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0484732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0485177Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0485606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0486011Z return func(*args, **kwargs) 2025-12-04T09:43:50.0486434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0486897Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0487384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0487911Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0488120Z 2025-12-04T09:43:50.0488236Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0488627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0488955Z res = mod(**inputs) 2025-12-04T09:43:50.0489324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0489722Z outputs = self.model( 2025-12-04T09:43:50.0492553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0492985Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0493385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0493809Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0494182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0494561Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0494964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0495375Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0495774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0496157Z return func(*args, **kwargs) 2025-12-04T09:43:50.0496580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0497001Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0497154Z 2025-12-04T09:43:50.0497265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0497640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0497961Z res = mod(**inputs) 2025-12-04T09:43:50.0498332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0498722Z outputs = self.model( 2025-12-04T09:43:50.0499099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0499488Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0499878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0500279Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0500640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0501008Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0501408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0501846Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0502023Z 2025-12-04T09:43:50.0502129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0502496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0502826Z res = mod(**inputs) 2025-12-04T09:43:50.0503198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0503602Z outputs = self.model( 2025-12-04T09:43:50.0503983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0504383Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0504766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0505162Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0505521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0505896Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0506287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0506732Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0506909Z 2025-12-04T09:43:50.0507100Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0507476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0507804Z res = mod(**inputs) 2025-12-04T09:43:50.0508181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0508558Z outputs = self.model( 2025-12-04T09:43:50.0508910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0509292Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0509666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0510045Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0510385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0510763Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0511148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0511532Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0511671Z 2025-12-04T09:43:50.0511773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0512124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0512440Z res = mod(**inputs) 2025-12-04T09:43:50.0512789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0513168Z outputs = self.model( 2025-12-04T09:43:50.0513532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0513920Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0514293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0514674Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0515025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0515375Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0515759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:43:50.0516150Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.0516282Z 2025-12-04T09:43:50.0516392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0516738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0517058Z res = mod(**inputs) 2025-12-04T09:43:50.0517411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0517780Z outputs = self.model( 2025-12-04T09:43:50.0518141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0518520Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0518894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0519270Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0519616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0519982Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0520496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0520953Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0521404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0521830Z return func(*args, **kwargs) 2025-12-04T09:43:50.0522243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0522705Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0522916Z 2025-12-04T09:43:50.0523023Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0523398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0523735Z res = mod(**inputs) 2025-12-04T09:43:50.0524129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0524558Z outputs = self.model( 2025-12-04T09:43:50.0524952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0525361Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0525780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0526191Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0526556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0526956Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0527385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0527823Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0528296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0528706Z return func(*args, **kwargs) 2025-12-04T09:43:50.0529108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0529525Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0529677Z 2025-12-04T09:43:50.0529789Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0530176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0530518Z res = mod(**inputs) 2025-12-04T09:43:50.0530899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0531308Z outputs = self.model( 2025-12-04T09:43:50.0531702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0532116Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0532519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0532936Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0533303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0533664Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0534064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0534474Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0534877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0535289Z return func(*args, **kwargs) 2025-12-04T09:43:50.0535691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0536166Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0536312Z 2025-12-04T09:43:50.0536403Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0536659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0537028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0537359Z res = mod(**inputs) 2025-12-04T09:43:50.0537744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0538153Z outputs = self.model( 2025-12-04T09:43:50.0538528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0538967Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0539348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0539738Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0540095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0540458Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0540854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0541261Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0541664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0542044Z return func(*args, **kwargs) 2025-12-04T09:43:50.0542425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0542849Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0543319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0543838Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0544051Z 2025-12-04T09:43:50.0544158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0544525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0544849Z res = mod(**inputs) 2025-12-04T09:43:50.0545244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0545657Z outputs = self.model( 2025-12-04T09:43:50.0546057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0546446Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0546836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0547395Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0547759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0548141Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0548546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0548964Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0549363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0549807Z return func(*args, **kwargs) 2025-12-04T09:43:50.0550195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0550633Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0550774Z 2025-12-04T09:43:50.0550881Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0551281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0551618Z res = mod(**inputs) 2025-12-04T09:43:50.0551983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0552379Z outputs = self.model( 2025-12-04T09:43:50.0552759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0553157Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0553564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0553952Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0554315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0554678Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0555053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0555472Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0555637Z 2025-12-04T09:43:50.0555745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0556084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0556399Z res = mod(**inputs) 2025-12-04T09:43:50.0556748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0557126Z outputs = self.model( 2025-12-04T09:43:50.0557483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0557863Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0558240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0558611Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0558970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0559340Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0559731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0560159Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0560341Z 2025-12-04T09:43:50.0560522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0560894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0561228Z res = mod(**inputs) 2025-12-04T09:43:50.0561616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0562030Z outputs = self.model( 2025-12-04T09:43:50.0562430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0562811Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0563195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0563579Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0563951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0564309Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0564710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0565111Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0565252Z 2025-12-04T09:43:50.0565358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0565717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0566033Z res = mod(**inputs) 2025-12-04T09:43:50.0566391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0566759Z outputs = self.model( 2025-12-04T09:43:50.0567120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0567525Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0567902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0568289Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0568642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0569019Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0569419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0569930Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0570340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0570728Z return func(*args, **kwargs) 2025-12-04T09:43:50.0571112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0571592Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0571801Z 2025-12-04T09:43:50.0571922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0572288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0572609Z res = mod(**inputs) 2025-12-04T09:43:50.0572981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0573375Z outputs = self.model( 2025-12-04T09:43:50.0573743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0574144Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0574535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0574612Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0574848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0574934Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0575189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0575290Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0575538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0575610Z return func(*args, **kwargs) 2025-12-04T09:43:50.0575893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0575980Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0575984Z 2025-12-04T09:43:50.0576116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0576318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0576400Z res = mod(**inputs) 2025-12-04T09:43:50.0576667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0576736Z outputs = self.model( 2025-12-04T09:43:50.0576997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0577073Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0577325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0577434Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0577659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0577741Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0578001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0578093Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0578343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0578414Z return func(*args, **kwargs) 2025-12-04T09:43:50.0578665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0578759Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0578762Z 2025-12-04T09:43:50.0578849Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0578961Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0579164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0579230Z res = mod(**inputs) 2025-12-04T09:43:50.0579494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0579562Z outputs = self.model( 2025-12-04T09:43:50.0579814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0579896Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0580149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0580226Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0580451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0580532Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0580791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0580884Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0581130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0581207Z return func(*args, **kwargs) 2025-12-04T09:43:50.0581457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0581566Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0581860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0582014Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0582018Z 2025-12-04T09:43:50.0582134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0582352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0582441Z res = mod(**inputs) 2025-12-04T09:43:50.0582705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0582776Z outputs = self.model( 2025-12-04T09:43:50.0583043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0583119Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0583374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0583471Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0583706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0583795Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0584058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0584149Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0584416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0584485Z return func(*args, **kwargs) 2025-12-04T09:43:50.0584751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0584831Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0584834Z 2025-12-04T09:43:50.0584939Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0585146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0585211Z res = mod(**inputs) 2025-12-04T09:43:50.0585470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0585547Z outputs = self.model( 2025-12-04T09:43:50.0585808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0585887Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0586139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0586211Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0586443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0586524Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0586786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0586908Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0586912Z 2025-12-04T09:43:50.0587017Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0587221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0587284Z res = mod(**inputs) 2025-12-04T09:43:50.0587543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0587617Z outputs = self.model( 2025-12-04T09:43:50.0587875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0587973Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0588220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0588329Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0588573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0588652Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0588902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0589027Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0589030Z 2025-12-04T09:43:50.0589132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0589332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0589398Z res = mod(**inputs) 2025-12-04T09:43:50.0589670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0589746Z outputs = self.model( 2025-12-04T09:43:50.0589994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0590075Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0590329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0590400Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0590627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0590703Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0590950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0591039Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0591042Z 2025-12-04T09:43:50.0591145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0591348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0591415Z res = mod(**inputs) 2025-12-04T09:43:50.0591665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0591741Z outputs = self.model( 2025-12-04T09:43:50.0591988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0592066Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0592312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0592387Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0592610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0592688Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0592935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:43:50.0593021Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.0593024Z 2025-12-04T09:43:50.0593125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0593329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0593390Z res = mod(**inputs) 2025-12-04T09:43:50.0593630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0593703Z outputs = self.model( 2025-12-04T09:43:50.0593960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0594052Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0594294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0594377Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0594595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0594669Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0594909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0595004Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0595239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0595331Z return func(*args, **kwargs) 2025-12-04T09:43:50.0595574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0595721Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0595726Z 2025-12-04T09:43:50.0595834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0596025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0596092Z res = mod(**inputs) 2025-12-04T09:43:50.0596342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0596409Z outputs = self.model( 2025-12-04T09:43:50.0596665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0596738Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0596982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0597059Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0597276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0597359Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0597602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0597691Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0597938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0598005Z return func(*args, **kwargs) 2025-12-04T09:43:50.0598253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0598340Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0598344Z 2025-12-04T09:43:50.0598446Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0598648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0598712Z res = mod(**inputs) 2025-12-04T09:43:50.0598967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0599043Z outputs = self.model( 2025-12-04T09:43:50.0599298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0599377Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0599648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0599722Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0599947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0600043Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0600315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0600481Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0600734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0600818Z return func(*args, **kwargs) 2025-12-04T09:43:50.0601112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0601207Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0601234Z 2025-12-04T09:43:50.0601338Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0601462Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0601684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0601763Z res = mod(**inputs) 2025-12-04T09:43:50.0602053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0602136Z outputs = self.model( 2025-12-04T09:43:50.0602395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0602470Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0602734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0602807Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0603043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0603124Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0603379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0603486Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0603723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0603792Z return func(*args, **kwargs) 2025-12-04T09:43:50.0604079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0604182Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0604488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0604627Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0604632Z 2025-12-04T09:43:50.0604735Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0604945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0605012Z res = mod(**inputs) 2025-12-04T09:43:50.0605281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0605350Z outputs = self.model( 2025-12-04T09:43:50.0605608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0605691Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0605964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0606043Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0606277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0606376Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0606655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0606749Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0606994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0607075Z return func(*args, **kwargs) 2025-12-04T09:43:50.0607334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0607423Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0607443Z 2025-12-04T09:43:50.0607547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0607742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0607813Z res = mod(**inputs) 2025-12-04T09:43:50.0608067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0608133Z outputs = self.model( 2025-12-04T09:43:50.0608394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0608465Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0608718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0608788Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0609010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0609096Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0609351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0609475Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0609478Z 2025-12-04T09:43:50.0609580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0609775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0609844Z res = mod(**inputs) 2025-12-04T09:43:50.0610098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0610164Z outputs = self.model( 2025-12-04T09:43:50.0610423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0610496Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0610749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0610822Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0611043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0611128Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0611376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0611491Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0611502Z 2025-12-04T09:43:50.0611605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0611817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0611889Z res = mod(**inputs) 2025-12-04T09:43:50.0612145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0612250Z outputs = self.model( 2025-12-04T09:43:50.0612525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0612598Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0612852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0612922Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0613140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0613226Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0613475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0613572Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0613585Z 2025-12-04T09:43:50.0613688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0613885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0613956Z res = mod(**inputs) 2025-12-04T09:43:50.0614207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0614275Z outputs = self.model( 2025-12-04T09:43:50.0614538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0614611Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0614877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0614950Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0615179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0615269Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0615520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0615610Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0615858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0615930Z return func(*args, **kwargs) 2025-12-04T09:43:50.0616184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0616333Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0616338Z 2025-12-04T09:43:50.0616441Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0616645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0616710Z res = mod(**inputs) 2025-12-04T09:43:50.0616968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0617035Z outputs = self.model( 2025-12-04T09:43:50.0617282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0617359Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0617607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0617677Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0617922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0618000Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0618269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0618372Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0618617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0618693Z return func(*args, **kwargs) 2025-12-04T09:43:50.0618940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0619018Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0619029Z 2025-12-04T09:43:50.0619131Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0619343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0619413Z res = mod(**inputs) 2025-12-04T09:43:50.0619666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0619732Z outputs = self.model( 2025-12-04T09:43:50.0619991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0620062Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0620318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0620389Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0620609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0620695Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0620942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0621032Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0621280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0621349Z return func(*args, **kwargs) 2025-12-04T09:43:50.0621604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0621689Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0621693Z 2025-12-04T09:43:50.0621773Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0621884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0622078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0622150Z res = mod(**inputs) 2025-12-04T09:43:50.0622403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0622469Z outputs = self.model( 2025-12-04T09:43:50.0622728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0622800Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0623048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0623126Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0623344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0623426Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0623691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0623783Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0624062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0624128Z return func(*args, **kwargs) 2025-12-04T09:43:50.0624387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0624491Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0624768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0624902Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0624906Z 2025-12-04T09:43:50.0625004Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0625191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0625276Z res = mod(**inputs) 2025-12-04T09:43:50.0625521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0625594Z outputs = self.model( 2025-12-04T09:43:50.0625839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0625909Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0626158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0626227Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0626443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0626530Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0626777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0626876Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0627117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0627187Z return func(*args, **kwargs) 2025-12-04T09:43:50.0627441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0627522Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0627525Z 2025-12-04T09:43:50.0627635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0627827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0627890Z res = mod(**inputs) 2025-12-04T09:43:50.0628155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0628223Z outputs = self.model( 2025-12-04T09:43:50.0628472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0628555Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0628795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0628871Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0629084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0629160Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0629408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0629544Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0629547Z 2025-12-04T09:43:50.0629657Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0629864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0629926Z res = mod(**inputs) 2025-12-04T09:43:50.0630200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0630269Z outputs = self.model( 2025-12-04T09:43:50.0630521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0630609Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0630855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0630933Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0631161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0631239Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0631493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0631606Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0631609Z 2025-12-04T09:43:50.0631719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0631911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0631975Z res = mod(**inputs) 2025-12-04T09:43:50.0632234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0632302Z outputs = self.model( 2025-12-04T09:43:50.0632552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0632632Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0632880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0632958Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0633175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0633251Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0633504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0633586Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0633590Z 2025-12-04T09:43:50.0633700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0633901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0633965Z res = mod(**inputs) 2025-12-04T09:43:50.0634232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0634301Z outputs = self.model( 2025-12-04T09:43:50.0634560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0634639Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0634892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0634972Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0635194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0635273Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0635555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:43:50.0635650Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.0635654Z 2025-12-04T09:43:50.0635757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0635973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0636040Z res = mod(**inputs) 2025-12-04T09:43:50.0636307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0636376Z outputs = self.model( 2025-12-04T09:43:50.0636636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0636716Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0636973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0637070Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0637295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0637376Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0637635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0637725Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0637974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0638054Z return func(*args, **kwargs) 2025-12-04T09:43:50.0638303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0638463Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0638466Z 2025-12-04T09:43:50.0638572Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0638774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0638848Z res = mod(**inputs) 2025-12-04T09:43:50.0639107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0639181Z outputs = self.model( 2025-12-04T09:43:50.0639436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0639509Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0639776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0639856Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0640092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0640186Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0640534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0640647Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0640926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0641005Z return func(*args, **kwargs) 2025-12-04T09:43:50.0641295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0641384Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0641387Z 2025-12-04T09:43:50.0642297Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0642535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0642625Z res = mod(**inputs) 2025-12-04T09:43:50.0642906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0642998Z outputs = self.model( 2025-12-04T09:43:50.0643266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0643349Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0643603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0643683Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0643916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0644016Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0644276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0644369Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0644617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0644698Z return func(*args, **kwargs) 2025-12-04T09:43:50.0644954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0645053Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0645057Z 2025-12-04T09:43:50.0645143Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0645256Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0645477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0645549Z res = mod(**inputs) 2025-12-04T09:43:50.0645831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0645905Z outputs = self.model( 2025-12-04T09:43:50.0646179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0646264Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0646534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0646611Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0646856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0646940Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0647352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0647451Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0647700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0647788Z return func(*args, **kwargs) 2025-12-04T09:43:50.0648053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0648160Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0648483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0648638Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0648642Z 2025-12-04T09:43:50.0648806Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0649016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0649106Z res = mod(**inputs) 2025-12-04T09:43:50.0649370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0649465Z outputs = self.model( 2025-12-04T09:43:50.0649732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0649806Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0650064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0650146Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0650375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0650481Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0650752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0650847Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0651108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0651179Z return func(*args, **kwargs) 2025-12-04T09:43:50.0651445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0651537Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0651540Z 2025-12-04T09:43:50.0651647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0651860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0651929Z res = mod(**inputs) 2025-12-04T09:43:50.0652196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0652274Z outputs = self.model( 2025-12-04T09:43:50.0652539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0652614Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0652885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0652959Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0653199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0653277Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0653540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0653672Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0653675Z 2025-12-04T09:43:50.0653783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0653995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0654061Z res = mod(**inputs) 2025-12-04T09:43:50.0654331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0654405Z outputs = self.model( 2025-12-04T09:43:50.0654668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0654740Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0655032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0655108Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0655336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0655430Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0655705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0655833Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0655837Z 2025-12-04T09:43:50.0655942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0656149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0656212Z res = mod(**inputs) 2025-12-04T09:43:50.0656519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0656614Z outputs = self.model( 2025-12-04T09:43:50.0656874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0656948Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0657211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0657282Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0657513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0657592Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0657845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0657934Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0657937Z 2025-12-04T09:43:50.0658044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0658245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0658318Z res = mod(**inputs) 2025-12-04T09:43:50.0658575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0658652Z outputs = self.model( 2025-12-04T09:43:50.0658904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0658988Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0659240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0659312Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0659536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0659618Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0659864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0659967Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0660221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0660290Z return func(*args, **kwargs) 2025-12-04T09:43:50.0660538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0660686Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0660690Z 2025-12-04T09:43:50.0660798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0661003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0661069Z res = mod(**inputs) 2025-12-04T09:43:50.0661323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0661406Z outputs = self.model( 2025-12-04T09:43:50.0661682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0661756Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0662004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0662081Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0662296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0662373Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0662628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0662747Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0662997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0663066Z return func(*args, **kwargs) 2025-12-04T09:43:50.0663325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0663409Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0663413Z 2025-12-04T09:43:50.0663513Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0663712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0663773Z res = mod(**inputs) 2025-12-04T09:43:50.0664021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0664094Z outputs = self.model( 2025-12-04T09:43:50.0664337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0664408Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0664658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0664727Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0664948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0665023Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0665265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0665360Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0665598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0665666Z return func(*args, **kwargs) 2025-12-04T09:43:50.0665914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0665999Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0666002Z 2025-12-04T09:43:50.0666087Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0666187Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0666377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0666446Z res = mod(**inputs) 2025-12-04T09:43:50.0666694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0666768Z outputs = self.model( 2025-12-04T09:43:50.0667028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0667114Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0667363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0667450Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0667662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0667745Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0667995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0668090Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0668324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0668418Z return func(*args, **kwargs) 2025-12-04T09:43:50.0668661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0668759Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0669050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0669181Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0669184Z 2025-12-04T09:43:50.0669284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0669479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0669541Z res = mod(**inputs) 2025-12-04T09:43:50.0669786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0669859Z outputs = self.model( 2025-12-04T09:43:50.0670103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0670180Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0670422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0670490Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0670710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0670785Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0671026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:43:50.0671119Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:43:50.0671356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0671432Z return func(*args, **kwargs) 2025-12-04T09:43:50.0671675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0671754Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0671757Z 2025-12-04T09:43:50.0671865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0672052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0672120Z res = mod(**inputs) 2025-12-04T09:43:50.0672360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0672425Z outputs = self.model( 2025-12-04T09:43:50.0672689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0672763Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0673020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0673095Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0673323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0673409Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0673649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0673761Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0673764Z 2025-12-04T09:43:50.0673871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0674062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0674147Z res = mod(**inputs) 2025-12-04T09:43:50.0674427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0674499Z outputs = self.model( 2025-12-04T09:43:50.0674770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0674843Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0675118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0675195Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0675417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0675505Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0675774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:43:50.0675893Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0675898Z 2025-12-04T09:43:50.0676009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0676215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0676287Z res = mod(**inputs) 2025-12-04T09:43:50.0676553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0676619Z outputs = self.model( 2025-12-04T09:43:50.0676913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0676991Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0677279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0677364Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0677608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0677697Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0677985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:43:50.0678071Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0678076Z 2025-12-04T09:43:50.0678192Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0678408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0678482Z res = mod(**inputs) 2025-12-04T09:43:50.0678781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0678856Z outputs = self.model( 2025-12-04T09:43:50.0679136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:43:50.0679229Z encoder_outputs = self.encoder( 2025-12-04T09:43:50.0679516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:43:50.0679602Z layer_outputs = encoder_layer( 2025-12-04T09:43:50.0679838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0679929Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0680228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:43:50.0680315Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.0680338Z 2025-12-04T09:43:50.0680524Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0680747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0680821Z res = mod(**inputs) 2025-12-04T09:43:50.0681106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0681181Z outputs = self.model( 2025-12-04T09:43:50.0681472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0681552Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0681836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1098, in forward 2025-12-04T09:43:50.0682029Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-12-04T09:43:50.0682300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:43:50.0682387Z return func(*args, **kwargs) 2025-12-04T09:43:50.0682658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:43:50.0682878Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:43:50.0683212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T09:43:50.0683408Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:43:50.0683413Z 2025-12-04T09:43:50.0683526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0683730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0683796Z res = mod(**inputs) 2025-12-04T09:43:50.0684067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0684136Z outputs = self.model( 2025-12-04T09:43:50.0684396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0684475Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0684732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1098, in forward 2025-12-04T09:43:50.0684907Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-12-04T09:43:50.0685148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:43:50.0685222Z return func(*args, **kwargs) 2025-12-04T09:43:50.0685512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:43:50.0685725Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:43:50.0686082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T09:43:50.0686272Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:43:50.0686276Z 2025-12-04T09:43:50.0686382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0686590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0686653Z res = mod(**inputs) 2025-12-04T09:43:50.0686930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0687017Z outputs = self.model( 2025-12-04T09:43:50.0687271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0687356Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0687611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0687684Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0687915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0687996Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0688249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0688320Z return func(*args, **kwargs) 2025-12-04T09:43:50.0688576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0688690Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0688937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0689010Z return func(*args, **kwargs) 2025-12-04T09:43:50.0689275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0689430Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0689433Z 2025-12-04T09:43:50.0689546Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0689744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0689808Z res = mod(**inputs) 2025-12-04T09:43:50.0690073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0690146Z outputs = self.model( 2025-12-04T09:43:50.0690417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0690490Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0690739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0690818Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0691045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0691125Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0691365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0691449Z return func(*args, **kwargs) 2025-12-04T09:43:50.0691697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0691810Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0692073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0692150Z return func(*args, **kwargs) 2025-12-04T09:43:50.0692392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0692479Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0692482Z 2025-12-04T09:43:50.0692584Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0692778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0692846Z res = mod(**inputs) 2025-12-04T09:43:50.0693108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0693174Z outputs = self.model( 2025-12-04T09:43:50.0693422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0693494Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0693742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0693811Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0694023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0694108Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0694339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0694414Z return func(*args, **kwargs) 2025-12-04T09:43:50.0694662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0694759Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0695005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0695072Z return func(*args, **kwargs) 2025-12-04T09:43:50.0695317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0695409Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0695413Z 2025-12-04T09:43:50.0695493Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0695601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0695797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0695861Z res = mod(**inputs) 2025-12-04T09:43:50.0696116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0696183Z outputs = self.model( 2025-12-04T09:43:50.0696433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0696512Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0696761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0696840Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0697063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0697140Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0697408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0697482Z return func(*args, **kwargs) 2025-12-04T09:43:50.0697757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0697870Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0698113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0698189Z return func(*args, **kwargs) 2025-12-04T09:43:50.0698444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0698543Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0698852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0699000Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0699004Z 2025-12-04T09:43:50.0699115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0699313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0699377Z res = mod(**inputs) 2025-12-04T09:43:50.0699639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0699706Z outputs = self.model( 2025-12-04T09:43:50.0699965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0700036Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0700287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0700367Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0700589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0700668Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0700918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0700987Z return func(*args, **kwargs) 2025-12-04T09:43:50.0701253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0701347Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0701578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0701651Z return func(*args, **kwargs) 2025-12-04T09:43:50.0701892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0701973Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0701984Z 2025-12-04T09:43:50.0702084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0702275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0702344Z res = mod(**inputs) 2025-12-04T09:43:50.0702590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0702655Z outputs = self.model( 2025-12-04T09:43:50.0702905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0702975Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0703252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0703325Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0703544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0703642Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0703896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0703965Z return func(*args, **kwargs) 2025-12-04T09:43:50.0704224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0704333Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0704581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0704649Z return func(*args, **kwargs) 2025-12-04T09:43:50.0704918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0705075Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0705080Z 2025-12-04T09:43:50.0705184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0705389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0705454Z res = mod(**inputs) 2025-12-04T09:43:50.0705703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0705777Z outputs = self.model( 2025-12-04T09:43:50.0706027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0706099Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0706362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0706434Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0706667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0706747Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0706989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0707064Z return func(*args, **kwargs) 2025-12-04T09:43:50.0707310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0707426Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0707665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0707736Z return func(*args, **kwargs) 2025-12-04T09:43:50.0707993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0708075Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0708078Z 2025-12-04T09:43:50.0708182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0708385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0708448Z res = mod(**inputs) 2025-12-04T09:43:50.0708709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0708776Z outputs = self.model( 2025-12-04T09:43:50.0709025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0709105Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0709388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0709477Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0709713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0709806Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0710052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0710118Z return func(*args, **kwargs) 2025-12-04T09:43:50.0710366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0710479Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0710720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0710810Z return func(*args, **kwargs) 2025-12-04T09:43:50.0711062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0711147Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0711151Z 2025-12-04T09:43:50.0711243Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0711345Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0711541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0711611Z res = mod(**inputs) 2025-12-04T09:43:50.0711868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0711942Z outputs = self.model( 2025-12-04T09:43:50.0712198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0712272Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0712532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0712606Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0712831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0712915Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0713160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0713233Z return func(*args, **kwargs) 2025-12-04T09:43:50.0713484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0713591Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0713845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0713912Z return func(*args, **kwargs) 2025-12-04T09:43:50.0714170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0714271Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0714561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0714700Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0714703Z 2025-12-04T09:43:50.0714805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0715009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0715073Z res = mod(**inputs) 2025-12-04T09:43:50.0715348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0715448Z outputs = self.model( 2025-12-04T09:43:50.0715706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0715791Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0716046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0716118Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0716343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0716419Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0716656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0716750Z return func(*args, **kwargs) 2025-12-04T09:43:50.0717012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0717117Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0717376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0717446Z return func(*args, **kwargs) 2025-12-04T09:43:50.0717719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0717800Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0717804Z 2025-12-04T09:43:50.0717913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0718129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0718198Z res = mod(**inputs) 2025-12-04T09:43:50.0718477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0718547Z outputs = self.model( 2025-12-04T09:43:50.0718814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0718894Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0719160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0719231Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0719472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0719552Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0719844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0719919Z return func(*args, **kwargs) 2025-12-04T09:43:50.0720208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0720411Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0720421Z 2025-12-04T09:43:50.0720541Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0720767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0720836Z res = mod(**inputs) 2025-12-04T09:43:50.0721110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0721193Z outputs = self.model( 2025-12-04T09:43:50.0721484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0721567Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0721888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0721981Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0722229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0722312Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0722559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0722640Z return func(*args, **kwargs) 2025-12-04T09:43:50.0722895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0723016Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0723043Z 2025-12-04T09:43:50.0723151Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0723350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0723439Z res = mod(**inputs) 2025-12-04T09:43:50.0723698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0723766Z outputs = self.model( 2025-12-04T09:43:50.0724029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0724101Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0724370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0724441Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0724662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0724749Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0724992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0725067Z return func(*args, **kwargs) 2025-12-04T09:43:50.0725347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.0725434Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0725438Z 2025-12-04T09:43:50.0725556Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0725769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0725833Z res = mod(**inputs) 2025-12-04T09:43:50.0726097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0726168Z outputs = self.model( 2025-12-04T09:43:50.0726431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0726506Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0726766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0726847Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0727070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0727150Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0727402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0727474Z return func(*args, **kwargs) 2025-12-04T09:43:50.0727750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0727856Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0728114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0728192Z return func(*args, **kwargs) 2025-12-04T09:43:50.0728461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0728614Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0728624Z 2025-12-04T09:43:50.0728728Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0728926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0728998Z res = mod(**inputs) 2025-12-04T09:43:50.0729254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0729336Z outputs = self.model( 2025-12-04T09:43:50.0729601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0729677Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0729942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0730013Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0730240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0730325Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0730569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0730637Z return func(*args, **kwargs) 2025-12-04T09:43:50.0730902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0731005Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0731261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0731330Z return func(*args, **kwargs) 2025-12-04T09:43:50.0731587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0731676Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0731679Z 2025-12-04T09:43:50.0731784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0731988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0732054Z res = mod(**inputs) 2025-12-04T09:43:50.0732316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0732390Z outputs = self.model( 2025-12-04T09:43:50.0732647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0732723Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0732986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0733059Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0733293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0733373Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0733618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0733715Z return func(*args, **kwargs) 2025-12-04T09:43:50.0733972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0734097Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0734358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0734430Z return func(*args, **kwargs) 2025-12-04T09:43:50.0734694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0734780Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0734784Z 2025-12-04T09:43:50.0734868Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0734980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0735181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0735272Z res = mod(**inputs) 2025-12-04T09:43:50.0735532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0735602Z outputs = self.model( 2025-12-04T09:43:50.0735864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0735939Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0736196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0736277Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0736502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0736590Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0736836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0736908Z return func(*args, **kwargs) 2025-12-04T09:43:50.0737173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0737273Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0737523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0737592Z return func(*args, **kwargs) 2025-12-04T09:43:50.0737849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0737954Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0738249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0738389Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0738400Z 2025-12-04T09:43:50.0738506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0738705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0738777Z res = mod(**inputs) 2025-12-04T09:43:50.0739035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0739104Z outputs = self.model( 2025-12-04T09:43:50.0739366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0739440Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0739702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0739813Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0740039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0740143Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0740407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0740479Z return func(*args, **kwargs) 2025-12-04T09:43:50.0740745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0740846Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0741099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0741177Z return func(*args, **kwargs) 2025-12-04T09:43:50.0741429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0741535Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0741538Z 2025-12-04T09:43:50.0741644Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0741849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0741914Z res = mod(**inputs) 2025-12-04T09:43:50.0742175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0742248Z outputs = self.model( 2025-12-04T09:43:50.0742504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0742575Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0742842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0742915Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0743147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0743224Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0743471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0743545Z return func(*args, **kwargs) 2025-12-04T09:43:50.0743798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 486, in forward 2025-12-04T09:43:50.0743880Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.0743883Z 2025-12-04T09:43:50.0743994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0744193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0744265Z res = mod(**inputs) 2025-12-04T09:43:50.0744524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0744592Z outputs = self.model( 2025-12-04T09:43:50.0744858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0744929Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0745193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0745264Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0745493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0745581Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0745850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0745923Z return func(*args, **kwargs) 2025-12-04T09:43:50.0746192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0746326Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0746608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0746691Z return func(*args, **kwargs) 2025-12-04T09:43:50.0746944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0747325Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0747331Z 2025-12-04T09:43:50.0747444Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0747652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0747754Z res = mod(**inputs) 2025-12-04T09:43:50.0748024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0748102Z outputs = self.model( 2025-12-04T09:43:50.0748354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0748427Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0748689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0748760Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0748987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0749065Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0749309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0749385Z return func(*args, **kwargs) 2025-12-04T09:43:50.0749640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0749747Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0749997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0750065Z return func(*args, **kwargs) 2025-12-04T09:43:50.0750323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0750403Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0750406Z 2025-12-04T09:43:50.0750508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0750714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0750779Z res = mod(**inputs) 2025-12-04T09:43:50.0751040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0751107Z outputs = self.model( 2025-12-04T09:43:50.0751361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0751440Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0751692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0751762Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0751992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0752071Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0752343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0752432Z return func(*args, **kwargs) 2025-12-04T09:43:50.0752687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0752824Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0753061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0753134Z return func(*args, **kwargs) 2025-12-04T09:43:50.0753384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0753467Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0753471Z 2025-12-04T09:43:50.0753557Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0753677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0753869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0753942Z res = mod(**inputs) 2025-12-04T09:43:50.0754196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0754270Z outputs = self.model( 2025-12-04T09:43:50.0754519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0754590Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0754852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0754921Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0755133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0755216Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0755449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0755525Z return func(*args, **kwargs) 2025-12-04T09:43:50.0755775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0755882Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0756129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0756195Z return func(*args, **kwargs) 2025-12-04T09:43:50.0756451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0756548Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0756834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0756976Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0756979Z 2025-12-04T09:43:50.0757086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0757279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0757350Z res = mod(**inputs) 2025-12-04T09:43:50.0757600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0757673Z outputs = self.model( 2025-12-04T09:43:50.0757923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0757994Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0758266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0758351Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0758586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0758680Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0758924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0758997Z return func(*args, **kwargs) 2025-12-04T09:43:50.0759245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0759351Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0759600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0759695Z return func(*args, **kwargs) 2025-12-04T09:43:50.0759951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0760031Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0760035Z 2025-12-04T09:43:50.0760137Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0760336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0760583Z res = mod(**inputs) 2025-12-04T09:43:50.0760841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0760916Z outputs = self.model( 2025-12-04T09:43:50.0761167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0761251Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0761502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0761581Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0761807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0761884Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0762155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0762223Z return func(*args, **kwargs) 2025-12-04T09:43:50.0762479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0762606Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0762610Z 2025-12-04T09:43:50.0762716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0762913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0762988Z res = mod(**inputs) 2025-12-04T09:43:50.0763240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0763320Z outputs = self.model( 2025-12-04T09:43:50.0763572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0763644Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0763898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0763967Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0764208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0764287Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0764519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0764610Z return func(*args, **kwargs) 2025-12-04T09:43:50.0764868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0764985Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0764989Z 2025-12-04T09:43:50.0765097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0765288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0765357Z res = mod(**inputs) 2025-12-04T09:43:50.0765601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0765686Z outputs = self.model( 2025-12-04T09:43:50.0765935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0766006Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0766255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0766325Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0766537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0766621Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0766853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0766920Z return func(*args, **kwargs) 2025-12-04T09:43:50.0767177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.0767262Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0767265Z 2025-12-04T09:43:50.0767376Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0767572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0767637Z res = mod(**inputs) 2025-12-04T09:43:50.0767892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0767957Z outputs = self.model( 2025-12-04T09:43:50.0768214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0768293Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0768535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0768611Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0768822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0768898Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0769139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0769205Z return func(*args, **kwargs) 2025-12-04T09:43:50.0769450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0769547Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0769776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0769851Z return func(*args, **kwargs) 2025-12-04T09:43:50.0770104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0770251Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0770278Z 2025-12-04T09:43:50.0770378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0770583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0770654Z res = mod(**inputs) 2025-12-04T09:43:50.0770899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0770965Z outputs = self.model( 2025-12-04T09:43:50.0771215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0771286Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0771536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0771622Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0771837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0771919Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0772152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0772219Z return func(*args, **kwargs) 2025-12-04T09:43:50.0772467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0772563Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0772799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0772868Z return func(*args, **kwargs) 2025-12-04T09:43:50.0773107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0773193Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0773196Z 2025-12-04T09:43:50.0773295Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0773491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0773553Z res = mod(**inputs) 2025-12-04T09:43:50.0773792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0773864Z outputs = self.model( 2025-12-04T09:43:50.0774104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0774175Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0774427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0774495Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0774715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0774790Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0775021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0775092Z return func(*args, **kwargs) 2025-12-04T09:43:50.0775336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0775429Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0775666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0775750Z return func(*args, **kwargs) 2025-12-04T09:43:50.0776001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0776098Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0776102Z 2025-12-04T09:43:50.0776195Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0776304Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0776497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0776564Z res = mod(**inputs) 2025-12-04T09:43:50.0776811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0776874Z outputs = self.model( 2025-12-04T09:43:50.0777127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0777213Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0777460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0777537Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0777756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0777836Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0778069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0778135Z return func(*args, **kwargs) 2025-12-04T09:43:50.0778386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0778480Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0778716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0778790Z return func(*args, **kwargs) 2025-12-04T09:43:50.0779037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0779139Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0779420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0779548Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0779551Z 2025-12-04T09:43:50.0779657Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0779850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0779919Z res = mod(**inputs) 2025-12-04T09:43:50.0780166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0780233Z outputs = self.model( 2025-12-04T09:43:50.0780486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0780558Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0780804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0780882Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0781098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0781178Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0781413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0781497Z return func(*args, **kwargs) 2025-12-04T09:43:50.0781745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0781854Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0782115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0782182Z return func(*args, **kwargs) 2025-12-04T09:43:50.0782428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0782514Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0782517Z 2025-12-04T09:43:50.0782614Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0782803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0782886Z res = mod(**inputs) 2025-12-04T09:43:50.0783149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0783222Z outputs = self.model( 2025-12-04T09:43:50.0783462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0783534Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0783783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0783851Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0784069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0784150Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0784376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0784451Z return func(*args, **kwargs) 2025-12-04T09:43:50.0784684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0784787Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0785020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0785085Z return func(*args, **kwargs) 2025-12-04T09:43:50.0785325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0785465Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0785468Z 2025-12-04T09:43:50.0785565Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0785758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0785823Z res = mod(**inputs) 2025-12-04T09:43:50.0786059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0786131Z outputs = self.model( 2025-12-04T09:43:50.0786369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0786445Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0786680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0786748Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0786962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0787036Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0787280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0787347Z return func(*args, **kwargs) 2025-12-04T09:43:50.0787595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0787719Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0787944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0788008Z return func(*args, **kwargs) 2025-12-04T09:43:50.0788248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0788323Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0788325Z 2025-12-04T09:43:50.0788429Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0788615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0788691Z res = mod(**inputs) 2025-12-04T09:43:50.0788939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0789003Z outputs = self.model( 2025-12-04T09:43:50.0789250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0789318Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0789558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0789633Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0789844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0789918Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0790154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0790220Z return func(*args, **kwargs) 2025-12-04T09:43:50.0790468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0790568Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0790798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0790868Z return func(*args, **kwargs) 2025-12-04T09:43:50.0791107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0791188Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0791197Z 2025-12-04T09:43:50.0791273Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0791372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0791570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0791633Z res = mod(**inputs) 2025-12-04T09:43:50.0791884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0791957Z outputs = self.model( 2025-12-04T09:43:50.0792205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0792281Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0792526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0792596Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0792840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0792918Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0793147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0793240Z return func(*args, **kwargs) 2025-12-04T09:43:50.0793500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0793611Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0793843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0793911Z return func(*args, **kwargs) 2025-12-04T09:43:50.0794160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0794254Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0794556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0794686Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0794689Z 2025-12-04T09:43:50.0794789Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0794988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0795050Z res = mod(**inputs) 2025-12-04T09:43:50.0795301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0795375Z outputs = self.model( 2025-12-04T09:43:50.0795625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0795704Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0795955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0796027Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0796251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0796331Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0796576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0796652Z return func(*args, **kwargs) 2025-12-04T09:43:50.0796900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0797013Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0797251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0797320Z return func(*args, **kwargs) 2025-12-04T09:43:50.0797578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0797659Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0797662Z 2025-12-04T09:43:50.0797773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0797967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0798029Z res = mod(**inputs) 2025-12-04T09:43:50.0798285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0798351Z outputs = self.model( 2025-12-04T09:43:50.0798599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0798698Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0798945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0799038Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0799270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0799348Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0799591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0799659Z return func(*args, **kwargs) 2025-12-04T09:43:50.0799914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 503, in forward 2025-12-04T09:43:50.0799994Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.0799998Z 2025-12-04T09:43:50.0800102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0800341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0800472Z res = mod(**inputs) 2025-12-04T09:43:50.0800732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0800810Z outputs = self.model( 2025-12-04T09:43:50.0801070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0801158Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0801435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0801514Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0801767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0801849Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0802092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0802171Z return func(*args, **kwargs) 2025-12-04T09:43:50.0802425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0802553Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0802558Z 2025-12-04T09:43:50.0802660Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0802858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0802931Z res = mod(**inputs) 2025-12-04T09:43:50.0803182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0803257Z outputs = self.model( 2025-12-04T09:43:50.0803515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0803589Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0803859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0803933Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0804160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0804246Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0804497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0804574Z return func(*args, **kwargs) 2025-12-04T09:43:50.0804894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0805014Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0805018Z 2025-12-04T09:43:50.0805159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0805355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0805442Z res = mod(**inputs) 2025-12-04T09:43:50.0805695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0805761Z outputs = self.model( 2025-12-04T09:43:50.0806021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0806092Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0806342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0806440Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0806662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0806751Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0806992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0807060Z return func(*args, **kwargs) 2025-12-04T09:43:50.0807314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.0807394Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0807397Z 2025-12-04T09:43:50.0807504Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0807696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0807759Z res = mod(**inputs) 2025-12-04T09:43:50.0808016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0808085Z outputs = self.model( 2025-12-04T09:43:50.0808334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0808415Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0808664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0808743Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0808961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0809037Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0809286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0809355Z return func(*args, **kwargs) 2025-12-04T09:43:50.0809602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0809710Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0809949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0810025Z return func(*args, **kwargs) 2025-12-04T09:43:50.0810273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0810421Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0810425Z 2025-12-04T09:43:50.0810534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0810746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0810821Z res = mod(**inputs) 2025-12-04T09:43:50.0811074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0811158Z outputs = self.model( 2025-12-04T09:43:50.0811432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0811507Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0811755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0811833Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0812052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0812136Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0812377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0812461Z return func(*args, **kwargs) 2025-12-04T09:43:50.0812719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0812820Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0813065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0813132Z return func(*args, **kwargs) 2025-12-04T09:43:50.0813382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0813467Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0813471Z 2025-12-04T09:43:50.0813573Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0813770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0813843Z res = mod(**inputs) 2025-12-04T09:43:50.0814095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0814169Z outputs = self.model( 2025-12-04T09:43:50.0814421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0814492Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0814754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0814824Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0815044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0815129Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0815377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0815453Z return func(*args, **kwargs) 2025-12-04T09:43:50.0815715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0815816Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0816069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0816138Z return func(*args, **kwargs) 2025-12-04T09:43:50.0816402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0816488Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0816491Z 2025-12-04T09:43:50.0816573Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0816701Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0816903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0816983Z res = mod(**inputs) 2025-12-04T09:43:50.0817245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0817327Z outputs = self.model( 2025-12-04T09:43:50.0817589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0817664Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0817917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0817996Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0818223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0818323Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0818566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0818637Z return func(*args, **kwargs) 2025-12-04T09:43:50.0818894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0818994Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0819235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0819313Z return func(*args, **kwargs) 2025-12-04T09:43:50.0819564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0819668Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0819966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0820101Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0820104Z 2025-12-04T09:43:50.0820214Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0820414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0820487Z res = mod(**inputs) 2025-12-04T09:43:50.0820745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0820813Z outputs = self.model( 2025-12-04T09:43:50.0821074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0821147Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0821403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0821482Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0821708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0821795Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0822040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0822110Z return func(*args, **kwargs) 2025-12-04T09:43:50.0822368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0822467Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0822711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0822804Z return func(*args, **kwargs) 2025-12-04T09:43:50.0823058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0823165Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0823168Z 2025-12-04T09:43:50.0823291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0823497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0823570Z res = mod(**inputs) 2025-12-04T09:43:50.0823829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0823905Z outputs = self.model( 2025-12-04T09:43:50.0824161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0824237Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0824518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0824592Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0824816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0824905Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0825157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0825234Z return func(*args, **kwargs) 2025-12-04T09:43:50.0825490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0825598Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0825848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0825919Z return func(*args, **kwargs) 2025-12-04T09:43:50.0826182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0826333Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0826338Z 2025-12-04T09:43:50.0826442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0826648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0826714Z res = mod(**inputs) 2025-12-04T09:43:50.0826970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0827046Z outputs = self.model( 2025-12-04T09:43:50.0827300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0827382Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0827636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0827710Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0827943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0828018Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0828246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0828319Z return func(*args, **kwargs) 2025-12-04T09:43:50.0828558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0828667Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0828924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0828991Z return func(*args, **kwargs) 2025-12-04T09:43:50.0829254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0829349Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0829352Z 2025-12-04T09:43:50.0829462Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0829653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0829715Z res = mod(**inputs) 2025-12-04T09:43:50.0829965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0830029Z outputs = self.model( 2025-12-04T09:43:50.0830274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0830367Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0830621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0830703Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0830925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0831002Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0831251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0831318Z return func(*args, **kwargs) 2025-12-04T09:43:50.0831579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0831682Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0831925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0831999Z return func(*args, **kwargs) 2025-12-04T09:43:50.0832255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0832339Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0832350Z 2025-12-04T09:43:50.0832428Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0832529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0832728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0832791Z res = mod(**inputs) 2025-12-04T09:43:50.0833047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0833122Z outputs = self.model( 2025-12-04T09:43:50.0833374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0833446Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0833709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0833779Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0834005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0834079Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0834320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0834393Z return func(*args, **kwargs) 2025-12-04T09:43:50.0834664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0834775Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0835004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0835085Z return func(*args, **kwargs) 2025-12-04T09:43:50.0835352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0835447Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0835727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0835862Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0835865Z 2025-12-04T09:43:50.0835963Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0836161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0836240Z res = mod(**inputs) 2025-12-04T09:43:50.0836485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0836561Z outputs = self.model( 2025-12-04T09:43:50.0836804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0836881Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0837128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0837199Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0837423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0837501Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0837743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0837819Z return func(*args, **kwargs) 2025-12-04T09:43:50.0838072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0838186Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0838424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0838491Z return func(*args, **kwargs) 2025-12-04T09:43:50.0838746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0838826Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0838829Z 2025-12-04T09:43:50.0838940Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0839143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0839208Z res = mod(**inputs) 2025-12-04T09:43:50.0839467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0839534Z outputs = self.model( 2025-12-04T09:43:50.0839785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0839866Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0840114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0840192Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0840481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0840585Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0840837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0840923Z return func(*args, **kwargs) 2025-12-04T09:43:50.0841194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0841324Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0841327Z 2025-12-04T09:43:50.0841432Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0841645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0841710Z res = mod(**inputs) 2025-12-04T09:43:50.0841960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0842038Z outputs = self.model( 2025-12-04T09:43:50.0842305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0842386Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0842636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0842709Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0842946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0843028Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0843287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0843368Z return func(*args, **kwargs) 2025-12-04T09:43:50.0843636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0843764Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0843767Z 2025-12-04T09:43:50.0843872Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0844079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0844157Z res = mod(**inputs) 2025-12-04T09:43:50.0844430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0844509Z outputs = self.model( 2025-12-04T09:43:50.0844777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0844854Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0845131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0845211Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0845452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0845544Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0845806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0845887Z return func(*args, **kwargs) 2025-12-04T09:43:50.0846158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.0846245Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0846249Z 2025-12-04T09:43:50.0846367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0846578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0846654Z res = mod(**inputs) 2025-12-04T09:43:50.0846946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0847020Z outputs = self.model( 2025-12-04T09:43:50.0847426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0847546Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0847820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0847905Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0848143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0848232Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0848494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0848608Z return func(*args, **kwargs) 2025-12-04T09:43:50.0848884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 512, in forward 2025-12-04T09:43:50.0848972Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.0848976Z 2025-12-04T09:43:50.0849090Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0849310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0849382Z res = mod(**inputs) 2025-12-04T09:43:50.0849664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0849736Z outputs = self.model( 2025-12-04T09:43:50.0850007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0850094Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0850371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0850456Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0850692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0850776Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0851043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0851117Z return func(*args, **kwargs) 2025-12-04T09:43:50.0851389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0851497Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0851737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0851813Z return func(*args, **kwargs) 2025-12-04T09:43:50.0852064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0852216Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0852221Z 2025-12-04T09:43:50.0852332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0852528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0852598Z res = mod(**inputs) 2025-12-04T09:43:50.0852853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0852919Z outputs = self.model( 2025-12-04T09:43:50.0853195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0853270Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0853519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0853619Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0853853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0853939Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0854179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0854246Z return func(*args, **kwargs) 2025-12-04T09:43:50.0854499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0854599Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0854854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0854929Z return func(*args, **kwargs) 2025-12-04T09:43:50.0855175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0855263Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0855266Z 2025-12-04T09:43:50.0855367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0855562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0855632Z res = mod(**inputs) 2025-12-04T09:43:50.0855880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0855954Z outputs = self.model( 2025-12-04T09:43:50.0856203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0856275Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0856531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0856604Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0856822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0856907Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0857142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0857216Z return func(*args, **kwargs) 2025-12-04T09:43:50.0857462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0857558Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0857805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0857874Z return func(*args, **kwargs) 2025-12-04T09:43:50.0858128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0858214Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0858218Z 2025-12-04T09:43:50.0858298Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0858406Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0858601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0858665Z res = mod(**inputs) 2025-12-04T09:43:50.0858924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0858990Z outputs = self.model( 2025-12-04T09:43:50.0859263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0859352Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0859617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0859700Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0859920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0859998Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0860245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0860311Z return func(*args, **kwargs) 2025-12-04T09:43:50.0860569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0860683Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0860924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0861002Z return func(*args, **kwargs) 2025-12-04T09:43:50.0861254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0861358Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0861653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0861781Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0861784Z 2025-12-04T09:43:50.0861890Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0862082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0862144Z res = mod(**inputs) 2025-12-04T09:43:50.0862400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0862466Z outputs = self.model( 2025-12-04T09:43:50.0862720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0862790Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0863040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0863117Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0863332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0863413Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0863651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0863720Z return func(*args, **kwargs) 2025-12-04T09:43:50.0863975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0864071Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0864304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0864377Z return func(*args, **kwargs) 2025-12-04T09:43:50.0864623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0864707Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0864710Z 2025-12-04T09:43:50.0864810Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0865018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0865090Z res = mod(**inputs) 2025-12-04T09:43:50.0865357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0865430Z outputs = self.model( 2025-12-04T09:43:50.0865689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0865775Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0866027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0866096Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0866312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0866398Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0866643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0866718Z return func(*args, **kwargs) 2025-12-04T09:43:50.0866959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0867064Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0867301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0867368Z return func(*args, **kwargs) 2025-12-04T09:43:50.0867608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0867757Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0867761Z 2025-12-04T09:43:50.0867864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0868062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0868127Z res = mod(**inputs) 2025-12-04T09:43:50.0868372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0868447Z outputs = self.model( 2025-12-04T09:43:50.0868690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0868769Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0869013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0869083Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0869305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0869382Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0869612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0869688Z return func(*args, **kwargs) 2025-12-04T09:43:50.0869929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0870040Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0870272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0870337Z return func(*args, **kwargs) 2025-12-04T09:43:50.0870583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0870661Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0870681Z 2025-12-04T09:43:50.0870791Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0870982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0871059Z res = mod(**inputs) 2025-12-04T09:43:50.0871338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0871404Z outputs = self.model( 2025-12-04T09:43:50.0871648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0871726Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0871966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0872041Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0872255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0872358Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0872594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0872663Z return func(*args, **kwargs) 2025-12-04T09:43:50.0872905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0873014Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0873246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0873317Z return func(*args, **kwargs) 2025-12-04T09:43:50.0873557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0873640Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0873644Z 2025-12-04T09:43:50.0873729Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0873827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0874022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0874085Z res = mod(**inputs) 2025-12-04T09:43:50.0874329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0874403Z outputs = self.model( 2025-12-04T09:43:50.0874648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0874718Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0874967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0875040Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0875259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0875336Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0875567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0875641Z return func(*args, **kwargs) 2025-12-04T09:43:50.0875882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0875990Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0876219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0876287Z return func(*args, **kwargs) 2025-12-04T09:43:50.0876558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0876657Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0876963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0877116Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0877120Z 2025-12-04T09:43:50.0877224Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0877426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0877489Z res = mod(**inputs) 2025-12-04T09:43:50.0877744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0877819Z outputs = self.model( 2025-12-04T09:43:50.0878065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0878159Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0878411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0878484Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0878717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0878802Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0879058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0879139Z return func(*args, **kwargs) 2025-12-04T09:43:50.0879406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0879525Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0879782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0879856Z return func(*args, **kwargs) 2025-12-04T09:43:50.0880130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0880218Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0880222Z 2025-12-04T09:43:50.0880340Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0880628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0880704Z res = mod(**inputs) 2025-12-04T09:43:50.0880992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0881069Z outputs = self.model( 2025-12-04T09:43:50.0881353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0881446Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0881733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0881822Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0882071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0882149Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0882398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0882468Z return func(*args, **kwargs) 2025-12-04T09:43:50.0882716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0882867Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0882871Z 2025-12-04T09:43:50.0882977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0883197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0883262Z res = mod(**inputs) 2025-12-04T09:43:50.0883530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0883606Z outputs = self.model( 2025-12-04T09:43:50.0883858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0883941Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0884190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0884262Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0884505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0884584Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0884821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0884897Z return func(*args, **kwargs) 2025-12-04T09:43:50.0885145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0885267Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0885270Z 2025-12-04T09:43:50.0885371Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0885564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0885636Z res = mod(**inputs) 2025-12-04T09:43:50.0885892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0885961Z outputs = self.model( 2025-12-04T09:43:50.0886217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0886290Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0886549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0886621Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0886837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0886922Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0887161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0887238Z return func(*args, **kwargs) 2025-12-04T09:43:50.0887485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.0887567Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0887570Z 2025-12-04T09:43:50.0887678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0887871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0887937Z res = mod(**inputs) 2025-12-04T09:43:50.0888195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0888261Z outputs = self.model( 2025-12-04T09:43:50.0888514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0888585Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0888851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0888946Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0889164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0889263Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0889504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0889572Z return func(*args, **kwargs) 2025-12-04T09:43:50.0889828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0889926Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0890164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0890256Z return func(*args, **kwargs) 2025-12-04T09:43:50.0890501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0890655Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0890658Z 2025-12-04T09:43:50.0890760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0890956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0891025Z res = mod(**inputs) 2025-12-04T09:43:50.0891272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0891345Z outputs = self.model( 2025-12-04T09:43:50.0891593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0891668Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0891921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0891992Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0892213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0892297Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0892533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0892606Z return func(*args, **kwargs) 2025-12-04T09:43:50.0892850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0892948Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0893199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0893267Z return func(*args, **kwargs) 2025-12-04T09:43:50.0893512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0893600Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0893603Z 2025-12-04T09:43:50.0893705Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0893903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0893966Z res = mod(**inputs) 2025-12-04T09:43:50.0894213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0894288Z outputs = self.model( 2025-12-04T09:43:50.0894553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0894634Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0894884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0894980Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0895240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0895318Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0895556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0895630Z return func(*args, **kwargs) 2025-12-04T09:43:50.0895877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0895980Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0896233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0896302Z return func(*args, **kwargs) 2025-12-04T09:43:50.0896559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0896648Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0896651Z 2025-12-04T09:43:50.0896739Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0896852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0897043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0897114Z res = mod(**inputs) 2025-12-04T09:43:50.0897362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0897428Z outputs = self.model( 2025-12-04T09:43:50.0897683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0897755Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0898007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0898079Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0898293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0898375Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0898608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0898675Z return func(*args, **kwargs) 2025-12-04T09:43:50.0898928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0899026Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0899270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0899338Z return func(*args, **kwargs) 2025-12-04T09:43:50.0899587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0899691Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0899980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0900121Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0900124Z 2025-12-04T09:43:50.0900229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0900440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0900515Z res = mod(**inputs) 2025-12-04T09:43:50.0900767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0900853Z outputs = self.model( 2025-12-04T09:43:50.0901155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0901230Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0901486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0901556Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0901788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0901872Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0902109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0902198Z return func(*args, **kwargs) 2025-12-04T09:43:50.0902442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0902539Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0902778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0902845Z return func(*args, **kwargs) 2025-12-04T09:43:50.0903088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0903173Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0903176Z 2025-12-04T09:43:50.0903275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0903472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0903536Z res = mod(**inputs) 2025-12-04T09:43:50.0903776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0903851Z outputs = self.model( 2025-12-04T09:43:50.0904099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0904169Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0904418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0904488Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0904706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0904781Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0905016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0905092Z return func(*args, **kwargs) 2025-12-04T09:43:50.0905336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 486, in forward 2025-12-04T09:43:50.0905421Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.0905424Z 2025-12-04T09:43:50.0905524Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0905711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0905782Z res = mod(**inputs) 2025-12-04T09:43:50.0906023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0906089Z outputs = self.model( 2025-12-04T09:43:50.0906353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0906426Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0906688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0906757Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0906984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0907068Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0907301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0907373Z return func(*args, **kwargs) 2025-12-04T09:43:50.0907617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0907720Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0907973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0908041Z return func(*args, **kwargs) 2025-12-04T09:43:50.0908283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0908433Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0908436Z 2025-12-04T09:43:50.0908534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0908730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0908791Z res = mod(**inputs) 2025-12-04T09:43:50.0909033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0909104Z outputs = self.model( 2025-12-04T09:43:50.0909348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0909425Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0909672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0909741Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0909966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0910043Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0910279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0910354Z return func(*args, **kwargs) 2025-12-04T09:43:50.0910603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0910716Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0910955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0911025Z return func(*args, **kwargs) 2025-12-04T09:43:50.0911283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0911362Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0911365Z 2025-12-04T09:43:50.0911483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0911672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0911732Z res = mod(**inputs) 2025-12-04T09:43:50.0911984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0912066Z outputs = self.model( 2025-12-04T09:43:50.0912312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0912406Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0912673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0912755Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0912983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0913060Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0913312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0913379Z return func(*args, **kwargs) 2025-12-04T09:43:50.0913634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0913764Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0914003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0914078Z return func(*args, **kwargs) 2025-12-04T09:43:50.0914387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0914473Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0914476Z 2025-12-04T09:43:50.0914565Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0914669Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0914869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0914932Z res = mod(**inputs) 2025-12-04T09:43:50.0915187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0915263Z outputs = self.model( 2025-12-04T09:43:50.0915515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0915589Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0915850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0915921Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0916145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0916220Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0916457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0916534Z return func(*args, **kwargs) 2025-12-04T09:43:50.0916845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0916952Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0917199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0917266Z return func(*args, **kwargs) 2025-12-04T09:43:50.0917520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0917615Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0917901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0918037Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0918065Z 2025-12-04T09:43:50.0918174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0918376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0918455Z res = mod(**inputs) 2025-12-04T09:43:50.0918726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0918804Z outputs = self.model( 2025-12-04T09:43:50.0919054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0919126Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0919386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0919457Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0919685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0919780Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0920018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0920095Z return func(*args, **kwargs) 2025-12-04T09:43:50.0920411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0920534Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0920771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0920839Z return func(*args, **kwargs) 2025-12-04T09:43:50.0921127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0921218Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0921224Z 2025-12-04T09:43:50.0921336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0921556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0921628Z res = mod(**inputs) 2025-12-04T09:43:50.0921913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0921987Z outputs = self.model( 2025-12-04T09:43:50.0922264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0922347Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0922605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0922687Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0922923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0923002Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0923250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0923320Z return func(*args, **kwargs) 2025-12-04T09:43:50.0923571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0923698Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0923702Z 2025-12-04T09:43:50.0923804Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0924008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0924072Z res = mod(**inputs) 2025-12-04T09:43:50.0924340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0924418Z outputs = self.model( 2025-12-04T09:43:50.0924666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0924757Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0925028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0925102Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0925330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0925408Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0925647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0925721Z return func(*args, **kwargs) 2025-12-04T09:43:50.0925984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0926109Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0926112Z 2025-12-04T09:43:50.0926215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0926410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0926482Z res = mod(**inputs) 2025-12-04T09:43:50.0926732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0926801Z outputs = self.model( 2025-12-04T09:43:50.0927059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0927131Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0927387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0927458Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0927675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0927763Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0928002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0928079Z return func(*args, **kwargs) 2025-12-04T09:43:50.0928331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.0928414Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0928417Z 2025-12-04T09:43:50.0928527Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0928723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0928788Z res = mod(**inputs) 2025-12-04T09:43:50.0929044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0929112Z outputs = self.model( 2025-12-04T09:43:50.0929368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0929441Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0929687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0929768Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0929984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0930062Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0930338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0930423Z return func(*args, **kwargs) 2025-12-04T09:43:50.0930691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0930804Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0931044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0931120Z return func(*args, **kwargs) 2025-12-04T09:43:50.0931371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0931527Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0931531Z 2025-12-04T09:43:50.0931635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0931849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0931924Z res = mod(**inputs) 2025-12-04T09:43:50.0932175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0932244Z outputs = self.model( 2025-12-04T09:43:50.0932499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0932571Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0932829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0932899Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0933116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0933204Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0933448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0933524Z return func(*args, **kwargs) 2025-12-04T09:43:50.0933775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0933872Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0934117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0934185Z return func(*args, **kwargs) 2025-12-04T09:43:50.0934435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0934521Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0934524Z 2025-12-04T09:43:50.0934630Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0934832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0934898Z res = mod(**inputs) 2025-12-04T09:43:50.0935151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0935226Z outputs = self.model( 2025-12-04T09:43:50.0935476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0935555Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0935807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0935878Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0936122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0936203Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0936441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0936533Z return func(*args, **kwargs) 2025-12-04T09:43:50.0936795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0936900Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0937140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0937207Z return func(*args, **kwargs) 2025-12-04T09:43:50.0937460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0937543Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0937569Z 2025-12-04T09:43:50.0937649Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0937757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0937951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0938022Z res = mod(**inputs) 2025-12-04T09:43:50.0938272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0938338Z outputs = self.model( 2025-12-04T09:43:50.0938592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0938663Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0938918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0938987Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0939210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0939295Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0939533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0939601Z return func(*args, **kwargs) 2025-12-04T09:43:50.0939857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0939954Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0940198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0940266Z return func(*args, **kwargs) 2025-12-04T09:43:50.0940512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0940619Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0940904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0941038Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0941051Z 2025-12-04T09:43:50.0941152Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0941347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0941418Z res = mod(**inputs) 2025-12-04T09:43:50.0941671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0941744Z outputs = self.model( 2025-12-04T09:43:50.0942042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0942127Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0942405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0942500Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0942764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0942858Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0943118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0943202Z return func(*args, **kwargs) 2025-12-04T09:43:50.0943456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0943554Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0943822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0943892Z return func(*args, **kwargs) 2025-12-04T09:43:50.0944140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0944232Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0944235Z 2025-12-04T09:43:50.0944337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0944539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0944603Z res = mod(**inputs) 2025-12-04T09:43:50.0944851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0944926Z outputs = self.model( 2025-12-04T09:43:50.0945173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0945247Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0945502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0945573Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0945799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0945876Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0946112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0946187Z return func(*args, **kwargs) 2025-12-04T09:43:50.0946435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0946547Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0946788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0946857Z return func(*args, **kwargs) 2025-12-04T09:43:50.0947276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0947431Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0947435Z 2025-12-04T09:43:50.0947539Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0947747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0947813Z res = mod(**inputs) 2025-12-04T09:43:50.0948074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0948143Z outputs = self.model( 2025-12-04T09:43:50.0948435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0948537Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0948794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0948890Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0949108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0949184Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0949421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0949488Z return func(*args, **kwargs) 2025-12-04T09:43:50.0949728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0949862Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0950093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0950170Z return func(*args, **kwargs) 2025-12-04T09:43:50.0950418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0950496Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0950500Z 2025-12-04T09:43:50.0950609Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0950796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0950858Z res = mod(**inputs) 2025-12-04T09:43:50.0951110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0951178Z outputs = self.model( 2025-12-04T09:43:50.0951426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0951497Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0951738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0951817Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0952028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0952109Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0952339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0952404Z return func(*args, **kwargs) 2025-12-04T09:43:50.0952652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0952756Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0952986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0953058Z return func(*args, **kwargs) 2025-12-04T09:43:50.0953297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0953385Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0953388Z 2025-12-04T09:43:50.0953466Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0953564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0953763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0953824Z res = mod(**inputs) 2025-12-04T09:43:50.0954079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0954153Z outputs = self.model( 2025-12-04T09:43:50.0954411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0954489Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0954743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0954814Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0955035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0955110Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0955350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0955417Z return func(*args, **kwargs) 2025-12-04T09:43:50.0955675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0955785Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0956016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0956082Z return func(*args, **kwargs) 2025-12-04T09:43:50.0956331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0956425Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0956718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0956851Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0956856Z 2025-12-04T09:43:50.0956958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0957162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0957228Z res = mod(**inputs) 2025-12-04T09:43:50.0957489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0957556Z outputs = self.model( 2025-12-04T09:43:50.0957809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0957891Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0958140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0958212Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0958440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0958521Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0958770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0958840Z return func(*args, **kwargs) 2025-12-04T09:43:50.0959089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0959200Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0959439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0959513Z return func(*args, **kwargs) 2025-12-04T09:43:50.0959762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0959843Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0959863Z 2025-12-04T09:43:50.0959973Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0960178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0960260Z res = mod(**inputs) 2025-12-04T09:43:50.0960618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0960692Z outputs = self.model( 2025-12-04T09:43:50.0960955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0961030Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0961286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0961376Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0961594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0961689Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0961940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0962018Z return func(*args, **kwargs) 2025-12-04T09:43:50.0962268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 503, in forward 2025-12-04T09:43:50.0962346Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.0962349Z 2025-12-04T09:43:50.0962449Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0962648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0962709Z res = mod(**inputs) 2025-12-04T09:43:50.0962962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0963030Z outputs = self.model( 2025-12-04T09:43:50.0963274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0963354Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0963598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0963666Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0963887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0963962Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0964199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0964266Z return func(*args, **kwargs) 2025-12-04T09:43:50.0964508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0964632Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0964636Z 2025-12-04T09:43:50.0964736Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0964933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0964996Z res = mod(**inputs) 2025-12-04T09:43:50.0965239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0965312Z outputs = self.model( 2025-12-04T09:43:50.0965559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0965632Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0965901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0965975Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0966214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0966291Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0966580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0966655Z return func(*args, **kwargs) 2025-12-04T09:43:50.0966897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.0967010Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.0967019Z 2025-12-04T09:43:50.0967120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0967313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0967398Z res = mod(**inputs) 2025-12-04T09:43:50.0967640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0967706Z outputs = self.model( 2025-12-04T09:43:50.0967954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0968022Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0968270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0968340Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0968549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0968631Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0968866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0968933Z return func(*args, **kwargs) 2025-12-04T09:43:50.0969183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.0969264Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.0969267Z 2025-12-04T09:43:50.0969383Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0969568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0969627Z res = mod(**inputs) 2025-12-04T09:43:50.0969871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0969935Z outputs = self.model( 2025-12-04T09:43:50.0970177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0970249Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0970490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0970565Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0970772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0970846Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0971079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0971143Z return func(*args, **kwargs) 2025-12-04T09:43:50.0971386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0971494Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0971723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0971814Z return func(*args, **kwargs) 2025-12-04T09:43:50.0972054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0972216Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0972219Z 2025-12-04T09:43:50.0972321Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0972509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0972577Z res = mod(**inputs) 2025-12-04T09:43:50.0972814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0972879Z outputs = self.model( 2025-12-04T09:43:50.0973144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0973214Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0973459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0973532Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0973745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0973828Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0974061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0974126Z return func(*args, **kwargs) 2025-12-04T09:43:50.0974376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0974474Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0974711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0974778Z return func(*args, **kwargs) 2025-12-04T09:43:50.0975021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0975105Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0975109Z 2025-12-04T09:43:50.0975208Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0975405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0975467Z res = mod(**inputs) 2025-12-04T09:43:50.0975710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0975785Z outputs = self.model( 2025-12-04T09:43:50.0976029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0976100Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0976349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0976418Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0976636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0976711Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0976940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0977015Z return func(*args, **kwargs) 2025-12-04T09:43:50.0977277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0977381Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0977612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0977693Z return func(*args, **kwargs) 2025-12-04T09:43:50.0977963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0978048Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0978051Z 2025-12-04T09:43:50.0978130Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0978235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0978426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0978494Z res = mod(**inputs) 2025-12-04T09:43:50.0978740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0978820Z outputs = self.model( 2025-12-04T09:43:50.0979068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0979141Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0979382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0979459Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0979673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0979759Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0979994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0980060Z return func(*args, **kwargs) 2025-12-04T09:43:50.0980324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0980420Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0980655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0980722Z return func(*args, **kwargs) 2025-12-04T09:43:50.0980963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0981062Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0981354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0981481Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0981492Z 2025-12-04T09:43:50.0981594Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0981776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0981845Z res = mod(**inputs) 2025-12-04T09:43:50.0982083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0982145Z outputs = self.model( 2025-12-04T09:43:50.0982388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0982456Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0982696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0982765Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0982987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0983071Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0983299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0983389Z return func(*args, **kwargs) 2025-12-04T09:43:50.0983644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.0983738Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.0983971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0984034Z return func(*args, **kwargs) 2025-12-04T09:43:50.0984270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0984353Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0984372Z 2025-12-04T09:43:50.0984471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0984664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0984725Z res = mod(**inputs) 2025-12-04T09:43:50.0984966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0985037Z outputs = self.model( 2025-12-04T09:43:50.0985270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0985337Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0985584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0985652Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0985869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0985945Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0986173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0986244Z return func(*args, **kwargs) 2025-12-04T09:43:50.0986484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0986586Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0986818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0986882Z return func(*args, **kwargs) 2025-12-04T09:43:50.0987124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.0987265Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.0987270Z 2025-12-04T09:43:50.0987369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0987564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0987623Z res = mod(**inputs) 2025-12-04T09:43:50.0987869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0987934Z outputs = self.model( 2025-12-04T09:43:50.0988171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0988247Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0988483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0988550Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0988781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0988883Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0989115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0989193Z return func(*args, **kwargs) 2025-12-04T09:43:50.0989431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0989541Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0989767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0989840Z return func(*args, **kwargs) 2025-12-04T09:43:50.0990084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.0990174Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.0990178Z 2025-12-04T09:43:50.0990284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0990470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0990533Z res = mod(**inputs) 2025-12-04T09:43:50.0990774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0990838Z outputs = self.model( 2025-12-04T09:43:50.0991081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0991149Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0991386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0991464Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0991672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0991745Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0991978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0992043Z return func(*args, **kwargs) 2025-12-04T09:43:50.0992287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0992388Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0992613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0992689Z return func(*args, **kwargs) 2025-12-04T09:43:50.0992928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.0993017Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.0993022Z 2025-12-04T09:43:50.0993098Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.0993197Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0993391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0993451Z res = mod(**inputs) 2025-12-04T09:43:50.0993688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0993758Z outputs = self.model( 2025-12-04T09:43:50.0993993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0994069Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0994321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0994392Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0994618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0994691Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0994939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0995005Z return func(*args, **kwargs) 2025-12-04T09:43:50.0995243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0995348Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0995574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0995656Z return func(*args, **kwargs) 2025-12-04T09:43:50.0995894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.0995987Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.0996274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.0996398Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.0996401Z 2025-12-04T09:43:50.0996498Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0996692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0996753Z res = mod(**inputs) 2025-12-04T09:43:50.0996996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0997062Z outputs = self.model( 2025-12-04T09:43:50.0997296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.0997375Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.0997615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.0997685Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.0997903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.0997980Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.0998217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0998284Z return func(*args, **kwargs) 2025-12-04T09:43:50.0998524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.0998634Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.0998868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.0998934Z return func(*args, **kwargs) 2025-12-04T09:43:50.0999183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.0999262Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.0999265Z 2025-12-04T09:43:50.0999373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.0999564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.0999628Z res = mod(**inputs) 2025-12-04T09:43:50.0999892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.0999959Z outputs = self.model( 2025-12-04T09:43:50.1000214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1000303Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1000628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1000713Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1000962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1001050Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1001328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1001406Z return func(*args, **kwargs) 2025-12-04T09:43:50.1001684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.1001824Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.1001829Z 2025-12-04T09:43:50.1001937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1002151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1002218Z res = mod(**inputs) 2025-12-04T09:43:50.1002499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1002581Z outputs = self.model( 2025-12-04T09:43:50.1002830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1002909Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1003155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1003227Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1003450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1003526Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1003768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1003834Z return func(*args, **kwargs) 2025-12-04T09:43:50.1004076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.1004196Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.1004199Z 2025-12-04T09:43:50.1004298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1004496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1004560Z res = mod(**inputs) 2025-12-04T09:43:50.1004805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1004878Z outputs = self.model( 2025-12-04T09:43:50.1005123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1005192Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1005444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1005513Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1005741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1005814Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1006069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1006148Z return func(*args, **kwargs) 2025-12-04T09:43:50.1006411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.1006502Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.1006515Z 2025-12-04T09:43:50.1006618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1006807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1006876Z res = mod(**inputs) 2025-12-04T09:43:50.1007118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1007183Z outputs = self.model( 2025-12-04T09:43:50.1007436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1007523Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1007777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1007848Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1008064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1008148Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1008385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1008451Z return func(*args, **kwargs) 2025-12-04T09:43:50.1008702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 512, in forward 2025-12-04T09:43:50.1008780Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.1008786Z 2025-12-04T09:43:50.1008892Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1009082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1009144Z res = mod(**inputs) 2025-12-04T09:43:50.1009397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1009462Z outputs = self.model( 2025-12-04T09:43:50.1009707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1009783Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1010026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1010100Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1010317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1010392Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1010633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1010701Z return func(*args, **kwargs) 2025-12-04T09:43:50.1010955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1011050Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1011283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1011354Z return func(*args, **kwargs) 2025-12-04T09:43:50.1011599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.1011759Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.1011771Z 2025-12-04T09:43:50.1011871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1012076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1012146Z res = mod(**inputs) 2025-12-04T09:43:50.1012411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1012478Z outputs = self.model( 2025-12-04T09:43:50.1012727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1012796Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1013043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1013113Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1013342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1013427Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1013660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1013728Z return func(*args, **kwargs) 2025-12-04T09:43:50.1013978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1014072Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1014311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1014379Z return func(*args, **kwargs) 2025-12-04T09:43:50.1014623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.1014710Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.1014713Z 2025-12-04T09:43:50.1014822Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1015016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1015077Z res = mod(**inputs) 2025-12-04T09:43:50.1015317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1015388Z outputs = self.model( 2025-12-04T09:43:50.1015625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1015694Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1015940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1016011Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1016229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1016303Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1016529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1016601Z return func(*args, **kwargs) 2025-12-04T09:43:50.1016837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1016929Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1017166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1017241Z return func(*args, **kwargs) 2025-12-04T09:43:50.1017502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.1017585Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.1017603Z 2025-12-04T09:43:50.1017680Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.1017785Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1017984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1018055Z res = mod(**inputs) 2025-12-04T09:43:50.1018293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1018357Z outputs = self.model( 2025-12-04T09:43:50.1018600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1018667Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1018904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1018994Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1019210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1019290Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1019522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1019588Z return func(*args, **kwargs) 2025-12-04T09:43:50.1019839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1019932Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1020163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1020239Z return func(*args, **kwargs) 2025-12-04T09:43:50.1020482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.1020585Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.1020864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.1020991Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.1020994Z 2025-12-04T09:43:50.1021100Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1021290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1021358Z res = mod(**inputs) 2025-12-04T09:43:50.1021603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1021669Z outputs = self.model( 2025-12-04T09:43:50.1021921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1021991Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1022235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1022309Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1022525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1022607Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1022839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1022905Z return func(*args, **kwargs) 2025-12-04T09:43:50.1023174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1023267Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1023497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1023576Z return func(*args, **kwargs) 2025-12-04T09:43:50.1023831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.1023916Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.1023919Z 2025-12-04T09:43:50.1024015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1024199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1024266Z res = mod(**inputs) 2025-12-04T09:43:50.1024507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1024601Z outputs = self.model( 2025-12-04T09:43:50.1024850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1024921Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1025175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1025244Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1025468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1025545Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1025779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1025855Z return func(*args, **kwargs) 2025-12-04T09:43:50.1026111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1026215Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1026455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1026519Z return func(*args, **kwargs) 2025-12-04T09:43:50.1026767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.1026908Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.1026911Z 2025-12-04T09:43:50.1027008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1027206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1027269Z res = mod(**inputs) 2025-12-04T09:43:50.1027522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1027588Z outputs = self.model( 2025-12-04T09:43:50.1027829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1027910Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1028152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1028221Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1028442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1028518Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1028761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1028827Z return func(*args, **kwargs) 2025-12-04T09:43:50.1029082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1029210Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1029456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1029525Z return func(*args, **kwargs) 2025-12-04T09:43:50.1029771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.1029847Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.1029851Z 2025-12-04T09:43:50.1029955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1030140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1030202Z res = mod(**inputs) 2025-12-04T09:43:50.1030467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1030531Z outputs = self.model( 2025-12-04T09:43:50.1030775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1030846Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1031081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1031155Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1031361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1031436Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1031671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1031738Z return func(*args, **kwargs) 2025-12-04T09:43:50.1031980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1032083Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1032311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1032382Z return func(*args, **kwargs) 2025-12-04T09:43:50.1032618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.1032704Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.1032708Z 2025-12-04T09:43:50.1032785Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.1032882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1033077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1033138Z res = mod(**inputs) 2025-12-04T09:43:50.1033375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1033449Z outputs = self.model( 2025-12-04T09:43:50.1033686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1033762Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1033998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1034066Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1034281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1055360Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1055939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1056030Z return func(*args, **kwargs) 2025-12-04T09:43:50.1056352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1056515Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1056770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1056841Z return func(*args, **kwargs) 2025-12-04T09:43:50.1057086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.1057180Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.1057461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.1057626Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.1057633Z 2025-12-04T09:43:50.1057745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1057940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1058002Z res = mod(**inputs) 2025-12-04T09:43:50.1058255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1058322Z outputs = self.model( 2025-12-04T09:43:50.1058567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1058637Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1058871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1058945Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1059168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1059248Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1059485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1059550Z return func(*args, **kwargs) 2025-12-04T09:43:50.1059805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1059910Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1060134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1060209Z return func(*args, **kwargs) 2025-12-04T09:43:50.1060449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.1060534Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.1060540Z 2025-12-04T09:43:50.1060642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1060838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1060910Z res = mod(**inputs) 2025-12-04T09:43:50.1061147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1061213Z outputs = self.model( 2025-12-04T09:43:50.1061467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1061538Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1061795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1061877Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1062089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1062184Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1062438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1062506Z return func(*args, **kwargs) 2025-12-04T09:43:50.1062746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.1062870Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.1062874Z 2025-12-04T09:43:50.1062975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1063170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1063247Z res = mod(**inputs) 2025-12-04T09:43:50.1063500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1063570Z outputs = self.model( 2025-12-04T09:43:50.1063813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1063884Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1064124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1064193Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1064411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1064487Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1064715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1064787Z return func(*args, **kwargs) 2025-12-04T09:43:50.1065029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.1065149Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.1065154Z 2025-12-04T09:43:50.1065252Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1065443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1065510Z res = mod(**inputs) 2025-12-04T09:43:50.1065751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1065820Z outputs = self.model( 2025-12-04T09:43:50.1066060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1066128Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1066372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1066438Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1066649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1066729Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1066959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1067027Z return func(*args, **kwargs) 2025-12-04T09:43:50.1067264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.1067340Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.1067345Z 2025-12-04T09:43:50.1067461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1067651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1067734Z res = mod(**inputs) 2025-12-04T09:43:50.1068002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1068067Z outputs = self.model( 2025-12-04T09:43:50.1068310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1068379Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1068615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1068689Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1068901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1069000Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1069233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1069303Z return func(*args, **kwargs) 2025-12-04T09:43:50.1069561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1069659Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1069884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1069957Z return func(*args, **kwargs) 2025-12-04T09:43:50.1070200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.1070360Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.1070365Z 2025-12-04T09:43:50.1070467Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1070663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1070734Z res = mod(**inputs) 2025-12-04T09:43:50.1070981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1071053Z outputs = self.model( 2025-12-04T09:43:50.1071294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1071361Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1071603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1071671Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1071884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1071964Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1072200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1072276Z return func(*args, **kwargs) 2025-12-04T09:43:50.1072522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1072617Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1072850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1072915Z return func(*args, **kwargs) 2025-12-04T09:43:50.1073175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.1073256Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.1073259Z 2025-12-04T09:43:50.1073357Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1073570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1073632Z res = mod(**inputs) 2025-12-04T09:43:50.1073888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1073961Z outputs = self.model( 2025-12-04T09:43:50.1074203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1074279Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1074519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1074589Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1074824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1074900Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1075131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1075205Z return func(*args, **kwargs) 2025-12-04T09:43:50.1075448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1075552Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1075783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1075849Z return func(*args, **kwargs) 2025-12-04T09:43:50.1076096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.1076179Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.1076183Z 2025-12-04T09:43:50.1076266Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.1076362Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1076551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1076616Z res = mod(**inputs) 2025-12-04T09:43:50.1076856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1076921Z outputs = self.model( 2025-12-04T09:43:50.1077171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1077241Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1077487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1077558Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1077771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1077853Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1078084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1078156Z return func(*args, **kwargs) 2025-12-04T09:43:50.1078396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1078489Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1078724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1078788Z return func(*args, **kwargs) 2025-12-04T09:43:50.1079046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.1079167Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.1079460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.1079595Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.1079599Z 2025-12-04T09:43:50.1079695Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1079880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1079947Z res = mod(**inputs) 2025-12-04T09:43:50.1080193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1080264Z outputs = self.model( 2025-12-04T09:43:50.1080632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1080718Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1080993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1081072Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1081310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1081401Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1081660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1081738Z return func(*args, **kwargs) 2025-12-04T09:43:50.1082012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1082130Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1082374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1082440Z return func(*args, **kwargs) 2025-12-04T09:43:50.1082686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.1082776Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.1082780Z 2025-12-04T09:43:50.1082878Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1083074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1083138Z res = mod(**inputs) 2025-12-04T09:43:50.1083384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1083461Z outputs = self.model( 2025-12-04T09:43:50.1083708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1083787Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1084032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1084103Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1084324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1084401Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1084636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1084713Z return func(*args, **kwargs) 2025-12-04T09:43:50.1084981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 486, in forward 2025-12-04T09:43:50.1085067Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.1085071Z 2025-12-04T09:43:50.1085189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1085386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1085471Z res = mod(**inputs) 2025-12-04T09:43:50.1085722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1085798Z outputs = self.model( 2025-12-04T09:43:50.1086054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1086123Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1086385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1086486Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1086708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1086793Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1087032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1087100Z return func(*args, **kwargs) 2025-12-04T09:43:50.1087354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1087459Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1087702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1087767Z return func(*args, **kwargs) 2025-12-04T09:43:50.1088017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.1088175Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.1088180Z 2025-12-04T09:43:50.1088280Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1088482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1088544Z res = mod(**inputs) 2025-12-04T09:43:50.1088800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1088874Z outputs = self.model( 2025-12-04T09:43:50.1089127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1089199Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1089451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1089523Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1089752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1089829Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1090068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1090144Z return func(*args, **kwargs) 2025-12-04T09:43:50.1090394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1090508Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1090747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1090828Z return func(*args, **kwargs) 2025-12-04T09:43:50.1091081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.1091182Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.1091186Z 2025-12-04T09:43:50.1091291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1091497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1091561Z res = mod(**inputs) 2025-12-04T09:43:50.1091814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1091880Z outputs = self.model( 2025-12-04T09:43:50.1092122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1092198Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1092456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1092533Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1092750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1092829Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1093068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1093138Z return func(*args, **kwargs) 2025-12-04T09:43:50.1093421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1093531Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1093759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1093832Z return func(*args, **kwargs) 2025-12-04T09:43:50.1094071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.1094155Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.1094158Z 2025-12-04T09:43:50.1094247Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.1094345Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1094536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1094597Z res = mod(**inputs) 2025-12-04T09:43:50.1094842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1094916Z outputs = self.model( 2025-12-04T09:43:50.1095162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1095233Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1095480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1095551Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1095769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1095854Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1096082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1096156Z return func(*args, **kwargs) 2025-12-04T09:43:50.1096393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1096502Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1096751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1096817Z return func(*args, **kwargs) 2025-12-04T09:43:50.1097078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.1097188Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.1097468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.1097605Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.1097608Z 2025-12-04T09:43:50.1097706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1097900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1097962Z res = mod(**inputs) 2025-12-04T09:43:50.1098220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1098292Z outputs = self.model( 2025-12-04T09:43:50.1098536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1098614Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1098853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1098922Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1099138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1099214Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1099438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1099508Z return func(*args, **kwargs) 2025-12-04T09:43:50.1099750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1099856Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1100078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1100142Z return func(*args, **kwargs) 2025-12-04T09:43:50.1100378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.1100455Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.1100458Z 2025-12-04T09:43:50.1100553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1100744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1100806Z res = mod(**inputs) 2025-12-04T09:43:50.1101050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1101116Z outputs = self.model( 2025-12-04T09:43:50.1101354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1101429Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1101662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1101733Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1101940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1102013Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1102286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1102352Z return func(*args, **kwargs) 2025-12-04T09:43:50.1102588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.1102720Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.1102724Z 2025-12-04T09:43:50.1102837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1103032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1103090Z res = mod(**inputs) 2025-12-04T09:43:50.1103327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1103392Z outputs = self.model( 2025-12-04T09:43:50.1103628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1103712Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1103949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1104015Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1104231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1104304Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1104529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1104602Z return func(*args, **kwargs) 2025-12-04T09:43:50.1104838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.1104956Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.1104960Z 2025-12-04T09:43:50.1105064Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1105257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1105328Z res = mod(**inputs) 2025-12-04T09:43:50.1105571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1105637Z outputs = self.model( 2025-12-04T09:43:50.1105887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1105959Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1106208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1106278Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1106493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1106601Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1106843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1106916Z return func(*args, **kwargs) 2025-12-04T09:43:50.1107152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.1107228Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.1107232Z 2025-12-04T09:43:50.1107335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1107519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1107580Z res = mod(**inputs) 2025-12-04T09:43:50.1107824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1107904Z outputs = self.model( 2025-12-04T09:43:50.1108146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1108230Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1108483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1108561Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1108768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1108840Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1109074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1109138Z return func(*args, **kwargs) 2025-12-04T09:43:50.1109382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1109491Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1109718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1109790Z return func(*args, **kwargs) 2025-12-04T09:43:50.1110026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.1110175Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.1110179Z 2025-12-04T09:43:50.1110275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1110459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1110526Z res = mod(**inputs) 2025-12-04T09:43:50.1110764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1110827Z outputs = self.model( 2025-12-04T09:43:50.1111067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1111135Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1111375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1111442Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1111648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1111731Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1111954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1112024Z return func(*args, **kwargs) 2025-12-04T09:43:50.1112265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1112358Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1112592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1112658Z return func(*args, **kwargs) 2025-12-04T09:43:50.1112891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.1112975Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.1112978Z 2025-12-04T09:43:50.1113075Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1113264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1113325Z res = mod(**inputs) 2025-12-04T09:43:50.1113584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1113659Z outputs = self.model( 2025-12-04T09:43:50.1113909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1113984Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1114234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1114303Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1114521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1114597Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1114826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1114901Z return func(*args, **kwargs) 2025-12-04T09:43:50.1115164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1115264Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1115491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1115556Z return func(*args, **kwargs) 2025-12-04T09:43:50.1115801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.1115883Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.1115886Z 2025-12-04T09:43:50.1115964Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.1116071Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1116261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1116332Z res = mod(**inputs) 2025-12-04T09:43:50.1116576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1116642Z outputs = self.model( 2025-12-04T09:43:50.1116888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1116958Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1117204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1117274Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1117487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1117567Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1117799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1117867Z return func(*args, **kwargs) 2025-12-04T09:43:50.1118116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1118212Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1118449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1118516Z return func(*args, **kwargs) 2025-12-04T09:43:50.1118757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.1118860Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.1119140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.1119286Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.1119297Z 2025-12-04T09:43:50.1119396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1119606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1119674Z res = mod(**inputs) 2025-12-04T09:43:50.1120203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1120273Z outputs = self.model( 2025-12-04T09:43:50.1120622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1120699Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1120962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1121040Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1121287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1121379Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1121634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1121705Z return func(*args, **kwargs) 2025-12-04T09:43:50.1121964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1122062Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1122312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1122380Z return func(*args, **kwargs) 2025-12-04T09:43:50.1122633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.1122725Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.1122728Z 2025-12-04T09:43:50.1122835Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1123047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1123113Z res = mod(**inputs) 2025-12-04T09:43:50.1123360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1123435Z outputs = self.model( 2025-12-04T09:43:50.1123681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1123752Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1124007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1124079Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1124299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1124376Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1124608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1124682Z return func(*args, **kwargs) 2025-12-04T09:43:50.1124923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1125037Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1125269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1125335Z return func(*args, **kwargs) 2025-12-04T09:43:50.1125599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.1125747Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.1125766Z 2025-12-04T09:43:50.1125868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1126079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1126143Z res = mod(**inputs) 2025-12-04T09:43:50.1126397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1126463Z outputs = self.model( 2025-12-04T09:43:50.1126704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1126781Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1127030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1127123Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1127343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1127420Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1127666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1127732Z return func(*args, **kwargs) 2025-12-04T09:43:50.1127981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1128093Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1128335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1128409Z return func(*args, **kwargs) 2025-12-04T09:43:50.1128670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.1128750Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.1128753Z 2025-12-04T09:43:50.1128863Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1129063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1129126Z res = mod(**inputs) 2025-12-04T09:43:50.1129393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1129468Z outputs = self.model( 2025-12-04T09:43:50.1129727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1129798Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1130049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1130126Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1130347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1130427Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1130668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1130734Z return func(*args, **kwargs) 2025-12-04T09:43:50.1130998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1131103Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1131348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1131437Z return func(*args, **kwargs) 2025-12-04T09:43:50.1131688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.1131797Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.1131800Z 2025-12-04T09:43:50.1131880Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.1131996Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1132196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1132260Z res = mod(**inputs) 2025-12-04T09:43:50.1132514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1132586Z outputs = self.model( 2025-12-04T09:43:50.1132841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1132950Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1133201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1133275Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1133506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1133584Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1133831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1133900Z return func(*args, **kwargs) 2025-12-04T09:43:50.1134153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1134267Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1134509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1134579Z return func(*args, **kwargs) 2025-12-04T09:43:50.1134843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.1134943Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.1135244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.1135379Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.1135383Z 2025-12-04T09:43:50.1135485Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1135690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1135753Z res = mod(**inputs) 2025-12-04T09:43:50.1136017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1136084Z outputs = self.model( 2025-12-04T09:43:50.1136343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1136425Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1136681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1136752Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1136982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1137058Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1137309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1137403Z return func(*args, **kwargs) 2025-12-04T09:43:50.1137655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1137784Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1138036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1138114Z return func(*args, **kwargs) 2025-12-04T09:43:50.1138365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.1138446Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.1138449Z 2025-12-04T09:43:50.1138558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1138751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1138816Z res = mod(**inputs) 2025-12-04T09:43:50.1139094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1139161Z outputs = self.model( 2025-12-04T09:43:50.1139416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1139489Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1139739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1139817Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1140036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1140112Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1140358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1140429Z return func(*args, **kwargs) 2025-12-04T09:43:50.1140686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 503, in forward 2025-12-04T09:43:50.1140767Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.1140771Z 2025-12-04T09:43:50.1140873Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1141074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1141139Z res = mod(**inputs) 2025-12-04T09:43:50.1141395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1141462Z outputs = self.model( 2025-12-04T09:43:50.1141713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1141793Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1142046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1142120Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1142347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1142427Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1142679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1142750Z return func(*args, **kwargs) 2025-12-04T09:43:50.1143005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.1143135Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.1143139Z 2025-12-04T09:43:50.1143257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1143471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1143556Z res = mod(**inputs) 2025-12-04T09:43:50.1143821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1143908Z outputs = self.model( 2025-12-04T09:43:50.1144158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1144228Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1144484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1144554Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1144779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1144874Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1145115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1145191Z return func(*args, **kwargs) 2025-12-04T09:43:50.1145439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.1145555Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.1145565Z 2025-12-04T09:43:50.1145667Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1145862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1145932Z res = mod(**inputs) 2025-12-04T09:43:50.1146182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1146251Z outputs = self.model( 2025-12-04T09:43:50.1146508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1146579Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1146834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1146906Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1147283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1147374Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1147613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1147682Z return func(*args, **kwargs) 2025-12-04T09:43:50.1147941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.1148024Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.1148028Z 2025-12-04T09:43:50.1148139Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1148333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1148399Z res = mod(**inputs) 2025-12-04T09:43:50.1148662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1148728Z outputs = self.model( 2025-12-04T09:43:50.1148986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1149067Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1149310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1149425Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1149638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1149740Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1150009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1150080Z return func(*args, **kwargs) 2025-12-04T09:43:50.1150405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1150515Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1150752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1150826Z return func(*args, **kwargs) 2025-12-04T09:43:50.1151078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.1151256Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.1151261Z 2025-12-04T09:43:50.1151363Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1151557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1151628Z res = mod(**inputs) 2025-12-04T09:43:50.1151874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1151954Z outputs = self.model( 2025-12-04T09:43:50.1152200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1152275Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1152523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1152592Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1152816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1152894Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1153135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1153202Z return func(*args, **kwargs) 2025-12-04T09:43:50.1153450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1153551Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1153787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1153853Z return func(*args, **kwargs) 2025-12-04T09:43:50.1154107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.1154185Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.1154188Z 2025-12-04T09:43:50.1154295Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1154488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1154550Z res = mod(**inputs) 2025-12-04T09:43:50.1154806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1154870Z outputs = self.model( 2025-12-04T09:43:50.1155124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1155192Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1155455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1155533Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1155762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1155859Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1156098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1156164Z return func(*args, **kwargs) 2025-12-04T09:43:50.1156413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1156505Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1156733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1156826Z return func(*args, **kwargs) 2025-12-04T09:43:50.1157075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.1157160Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.1157170Z 2025-12-04T09:43:50.1157249Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.1157350Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1157550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1157610Z res = mod(**inputs) 2025-12-04T09:43:50.1157861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1157933Z outputs = self.model( 2025-12-04T09:43:50.1158183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1158259Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1158504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1158574Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1158795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1158870Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1159108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1159181Z return func(*args, **kwargs) 2025-12-04T09:43:50.1159428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1159526Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1159764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1159831Z return func(*args, **kwargs) 2025-12-04T09:43:50.1160092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.1160189Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.1160551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.1160692Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.1160696Z 2025-12-04T09:43:50.1160798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1161010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1161076Z res = mod(**inputs) 2025-12-04T09:43:50.1161358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1161437Z outputs = self.model( 2025-12-04T09:43:50.1161708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1161790Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1162059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1162133Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1162364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1162452Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1162685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1162758Z return func(*args, **kwargs) 2025-12-04T09:43:50.1163016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:43:50.1163119Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:43:50.1163352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1163417Z return func(*args, **kwargs) 2025-12-04T09:43:50.1163670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.1163749Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.1163752Z 2025-12-04T09:43:50.1163859Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1164051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1164113Z res = mod(**inputs) 2025-12-04T09:43:50.1164366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1164431Z outputs = self.model( 2025-12-04T09:43:50.1164674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1164751Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1164992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1165067Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1165285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1165357Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1165591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1165657Z return func(*args, **kwargs) 2025-12-04T09:43:50.1165899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1166003Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1166229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1166300Z return func(*args, **kwargs) 2025-12-04T09:43:50.1166535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:43:50.1166674Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:43:50.1166684Z 2025-12-04T09:43:50.1166781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1166968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1167053Z res = mod(**inputs) 2025-12-04T09:43:50.1167299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1167380Z outputs = self.model( 2025-12-04T09:43:50.1167643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1167713Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1167967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1168037Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1168251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1168332Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1168568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1168650Z return func(*args, **kwargs) 2025-12-04T09:43:50.1168919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1169021Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1169260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1169323Z return func(*args, **kwargs) 2025-12-04T09:43:50.1169566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:43:50.1169649Z key_states = self.k_proj(current_states) 2025-12-04T09:43:50.1169652Z 2025-12-04T09:43:50.1169749Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1169945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1170006Z res = mod(**inputs) 2025-12-04T09:43:50.1170250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1170323Z outputs = self.model( 2025-12-04T09:43:50.1170570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1170639Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1170889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1170956Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1171174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1171247Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1171480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1171553Z return func(*args, **kwargs) 2025-12-04T09:43:50.1171793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1171894Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1172134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1172197Z return func(*args, **kwargs) 2025-12-04T09:43:50.1172443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:43:50.1172523Z value_states = self.v_proj(current_states) 2025-12-04T09:43:50.1172526Z 2025-12-04T09:43:50.1172603Z cudagraph partition due to non gpu ops 2025-12-04T09:43:50.1172720Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1172907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1172987Z res = mod(**inputs) 2025-12-04T09:43:50.1173223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1173301Z outputs = self.model( 2025-12-04T09:43:50.1173544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1173612Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1173849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1173923Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1174133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1174230Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1174457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1174523Z return func(*args, **kwargs) 2025-12-04T09:43:50.1174770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1174870Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1175098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1175167Z return func(*args, **kwargs) 2025-12-04T09:43:50.1175404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:43:50.1175502Z attn_output, attn_weights = attention_interface( 2025-12-04T09:43:50.1175785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:43:50.1175912Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:43:50.1175915Z 2025-12-04T09:43:50.1176081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1176270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1176338Z res = mod(**inputs) 2025-12-04T09:43:50.1176579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1176650Z outputs = self.model( 2025-12-04T09:43:50.1176891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1176959Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1177205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1177273Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1177489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1177563Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1177792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1177862Z return func(*args, **kwargs) 2025-12-04T09:43:50.1178097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:43:50.1178197Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:43:50.1178432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1178511Z return func(*args, **kwargs) 2025-12-04T09:43:50.1178756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:43:50.1178857Z attn_output = self.out_proj(attn_output) 2025-12-04T09:43:50.1178860Z 2025-12-04T09:43:50.1178975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1179171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1179231Z res = mod(**inputs) 2025-12-04T09:43:50.1179478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1179543Z outputs = self.model( 2025-12-04T09:43:50.1179781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1179856Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1180119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1180187Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1180401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1180476Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1180713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1180777Z return func(*args, **kwargs) 2025-12-04T09:43:50.1181013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.1181131Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.1181134Z 2025-12-04T09:43:50.1181229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1181422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1181481Z res = mod(**inputs) 2025-12-04T09:43:50.1181722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1181793Z outputs = self.model( 2025-12-04T09:43:50.1182031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1182099Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1182344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1182410Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1182623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1182698Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1182923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1182995Z return func(*args, **kwargs) 2025-12-04T09:43:50.1183235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:43:50.1183345Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:43:50.1183355Z 2025-12-04T09:43:50.1183451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1183637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1183705Z res = mod(**inputs) 2025-12-04T09:43:50.1183941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1184005Z outputs = self.model( 2025-12-04T09:43:50.1184267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1184349Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1184601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1184681Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1184888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1184968Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1185201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1185268Z return func(*args, **kwargs) 2025-12-04T09:43:50.1185520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:43:50.1185616Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:43:50.1185619Z 2025-12-04T09:43:50.1185724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1185922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1185981Z res = mod(**inputs) 2025-12-04T09:43:50.1186224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:43:50.1186288Z outputs = self.model( 2025-12-04T09:43:50.1186525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:43:50.1186598Z decoder_outputs = self.decoder( 2025-12-04T09:43:50.1186830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:43:50.1186905Z layer_outputs = decoder_layer( 2025-12-04T09:43:50.1187112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:43:50.1187184Z return super().__call__(*args, **kwargs) 2025-12-04T09:43:50.1187416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:43:50.1187480Z return func(*args, **kwargs) 2025-12-04T09:43:50.1187717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 512, in forward 2025-12-04T09:43:50.1187791Z hidden_states = residual + hidden_states 2025-12-04T09:43:50.1187794Z 2025-12-04T09:43:50.1187890Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1188079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1188140Z res = mod(**inputs) 2025-12-04T09:43:50.1188380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1419, in forward 2025-12-04T09:43:50.1188460Z lm_logits = self.lm_head(outputs[0]) 2025-12-04T09:43:50.1188465Z 2025-12-04T09:43:50.1188559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:50.1188749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:50.1188808Z res = mod(**inputs) 2025-12-04T09:43:50.1189044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1426, in forward 2025-12-04T09:43:50.1189208Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:43:50.1189212Z 2025-12-04T09:44:01.7212167Z Compilation time (from dynamo_timed): 26.454226249 2025-12-04T09:44:01.7309483Z pass 2025-12-04T09:44:01.7310561Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:01.7311724Z TIMING: _recursive_pre_grad_passes:0.01299 _recursive_joint_graph_passes:1.12488 _recursive_post_grad_passes:0.13026 async_compile.wait:0.76601 code_gen:11.28202 inductor_compile:14.32602 backend_compile:20.89898 gc:0.00063 entire_frame_compile:26.45423 total_wall_time:26.45423 2025-12-04T09:44:01.7313016Z STATS: call_* op count: 906 | FakeTensorMode.__torch_dispatch__:18114 | FakeTensor.__torch_dispatch__:10222 | ProxyTorchDispatchMode.__torch_dispatch__:4990 2025-12-04T09:44:01.7313545Z Dynamo produced 1 graphs covering 906 ops with 0 graph breaks (0 unique) 2025-12-04T09:44:04.4750758Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:44:04.4754221Z import pynvml # type: ignore[import] 2025-12-04T09:44:07.8423868Z 2025-12-04T09:44:10.1777909Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:44:10.1778212Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:44:10.1792846Z cpu eval MBartForCausalLM 2025-12-04T09:44:11.8773833Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:12.4896509Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:13.1312695Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:20.4449594Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4449911Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4450153Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4450383Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4450666Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4450993Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4451372Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4451739Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4452141Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4452458Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4452728Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4453153Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4453526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4454069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4454557Z res = mod(**inputs) 2025-12-04T09:44:20.4455123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4455657Z outputs = self.model.decoder( 2025-12-04T09:44:20.4456262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4456711Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4457117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4457548Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4457998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4458427Z return func(*args, **kwargs) 2025-12-04T09:44:20.4458837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4459344Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4459799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4460215Z return func(*args, **kwargs) 2025-12-04T09:44:20.4460936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4461466Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4461769Z 2025-12-04T09:44:20.4461937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4462395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4462779Z res = mod(**inputs) 2025-12-04T09:44:20.4463182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4463608Z outputs = self.model.decoder( 2025-12-04T09:44:20.4464028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4464453Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4464839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4465338Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4465750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4466159Z return func(*args, **kwargs) 2025-12-04T09:44:20.4466561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4467024Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4467458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4467852Z return func(*args, **kwargs) 2025-12-04T09:44:20.4468260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4468691Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4468843Z 2025-12-04T09:44:20.4468967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4469351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4469704Z res = mod(**inputs) 2025-12-04T09:44:20.4470103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4470550Z outputs = self.model.decoder( 2025-12-04T09:44:20.4470974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4471391Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4471769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4472189Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4472759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4473162Z return func(*args, **kwargs) 2025-12-04T09:44:20.4473563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4474100Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4474593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4474997Z return func(*args, **kwargs) 2025-12-04T09:44:20.4475394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4475840Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4476003Z 2025-12-04T09:44:20.4476090Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4476351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4476774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4477142Z res = mod(**inputs) 2025-12-04T09:44:20.4477562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4478004Z outputs = self.model.decoder( 2025-12-04T09:44:20.4478421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4478856Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4479247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4479651Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4480075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4480843Z return func(*args, **kwargs) 2025-12-04T09:44:20.4481266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4481721Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4482165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4482580Z return func(*args, **kwargs) 2025-12-04T09:44:20.4483125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4483707Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4484198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4484738Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4484944Z 2025-12-04T09:44:20.4485063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4485465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4485826Z res = mod(**inputs) 2025-12-04T09:44:20.4486245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4486676Z outputs = self.model.decoder( 2025-12-04T09:44:20.4487175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4487605Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4487988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4488391Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4488816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4489246Z return func(*args, **kwargs) 2025-12-04T09:44:20.4489657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4490113Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4490561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4490982Z return func(*args, **kwargs) 2025-12-04T09:44:20.4491389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4491830Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4491982Z 2025-12-04T09:44:20.4492106Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4492523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4492885Z res = mod(**inputs) 2025-12-04T09:44:20.4493285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4493744Z outputs = self.model.decoder( 2025-12-04T09:44:20.4494187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4494627Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4495020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4495415Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4495837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4496246Z return func(*args, **kwargs) 2025-12-04T09:44:20.4496660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4497169Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4497366Z 2025-12-04T09:44:20.4497476Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4497860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4498207Z res = mod(**inputs) 2025-12-04T09:44:20.4498591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4499026Z outputs = self.model.decoder( 2025-12-04T09:44:20.4499447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4499853Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4500236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4500625Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4501034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4501429Z return func(*args, **kwargs) 2025-12-04T09:44:20.4501830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4502293Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4502708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4503090Z return self.act(input) 2025-12-04T09:44:20.4503221Z 2025-12-04T09:44:20.4503336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4503738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4504089Z res = mod(**inputs) 2025-12-04T09:44:20.4504489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4504909Z outputs = self.model.decoder( 2025-12-04T09:44:20.4505316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4505724Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4506101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4506500Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4507120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4507549Z return func(*args, **kwargs) 2025-12-04T09:44:20.4507999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4508429Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4508584Z 2025-12-04T09:44:20.4508723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4509148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4509527Z res = mod(**inputs) 2025-12-04T09:44:20.4509923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4510361Z outputs = self.model.decoder( 2025-12-04T09:44:20.4510785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4511224Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4511600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4512007Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4512411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4512818Z return func(*args, **kwargs) 2025-12-04T09:44:20.4513215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4513660Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4514089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4514486Z return func(*args, **kwargs) 2025-12-04T09:44:20.4514887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4515391Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4515611Z 2025-12-04T09:44:20.4515737Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4516130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4516495Z res = mod(**inputs) 2025-12-04T09:44:20.4516882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4517297Z outputs = self.model.decoder( 2025-12-04T09:44:20.4517706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4518136Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4518523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4518941Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4519361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4519775Z return func(*args, **kwargs) 2025-12-04T09:44:20.4520184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4520818Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4521277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4521696Z return func(*args, **kwargs) 2025-12-04T09:44:20.4522110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4522559Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4522721Z 2025-12-04T09:44:20.4522839Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4523244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4523630Z res = mod(**inputs) 2025-12-04T09:44:20.4524031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4524485Z outputs = self.model.decoder( 2025-12-04T09:44:20.4524953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4525378Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4525767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4526181Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4526595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4527017Z return func(*args, **kwargs) 2025-12-04T09:44:20.4527436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4527916Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4528358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4528780Z return func(*args, **kwargs) 2025-12-04T09:44:20.4529198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4529636Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4529806Z 2025-12-04T09:44:20.4529895Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4530166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4530563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4530913Z res = mod(**inputs) 2025-12-04T09:44:20.4531314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4531751Z outputs = self.model.decoder( 2025-12-04T09:44:20.4532169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4532597Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4532984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4533387Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4533804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4534220Z return func(*args, **kwargs) 2025-12-04T09:44:20.4534641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4535099Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4535537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4535953Z return func(*args, **kwargs) 2025-12-04T09:44:20.4536371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4536822Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4537322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4537861Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4538068Z 2025-12-04T09:44:20.4538189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4538582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4538966Z res = mod(**inputs) 2025-12-04T09:44:20.4539372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4539842Z outputs = self.model.decoder( 2025-12-04T09:44:20.4540340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4540778Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4541165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4541569Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4541993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4542414Z return func(*args, **kwargs) 2025-12-04T09:44:20.4542832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4543295Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4543735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4544148Z return func(*args, **kwargs) 2025-12-04T09:44:20.4544580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4545020Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4545178Z 2025-12-04T09:44:20.4545292Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4545686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4546032Z res = mod(**inputs) 2025-12-04T09:44:20.4546429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4546862Z outputs = self.model.decoder( 2025-12-04T09:44:20.4547511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4548175Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4548635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4549024Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4549428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4549888Z return func(*args, **kwargs) 2025-12-04T09:44:20.4550369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4550846Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4551034Z 2025-12-04T09:44:20.4551148Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4551538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4551887Z res = mod(**inputs) 2025-12-04T09:44:20.4552273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4552693Z outputs = self.model.decoder( 2025-12-04T09:44:20.4553107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4553526Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4553898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4554325Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4554733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4555249Z return func(*args, **kwargs) 2025-12-04T09:44:20.4555651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4556152Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4556603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4556976Z return self.act(input) 2025-12-04T09:44:20.4557108Z 2025-12-04T09:44:20.4557223Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4557610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4557953Z res = mod(**inputs) 2025-12-04T09:44:20.4558329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4558743Z outputs = self.model.decoder( 2025-12-04T09:44:20.4559186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4559599Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4559976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4560438Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4560876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4561293Z return func(*args, **kwargs) 2025-12-04T09:44:20.4561712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4562160Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4562309Z 2025-12-04T09:44:20.4562430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4562813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4563154Z res = mod(**inputs) 2025-12-04T09:44:20.4563547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4563965Z outputs = self.model.decoder( 2025-12-04T09:44:20.4564374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4564790Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4565165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4565552Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4565962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4566366Z return func(*args, **kwargs) 2025-12-04T09:44:20.4566773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:44:20.4567203Z hidden_states = residual + hidden_states 2025-12-04T09:44:20.4567355Z 2025-12-04T09:44:20.4567467Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4567852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4568190Z res = mod(**inputs) 2025-12-04T09:44:20.4568576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4568996Z outputs = self.model.decoder( 2025-12-04T09:44:20.4569394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4569808Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4570204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4570601Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4571044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4571529Z return func(*args, **kwargs) 2025-12-04T09:44:20.4571953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4572401Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4572834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4573242Z return func(*args, **kwargs) 2025-12-04T09:44:20.4573653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4574165Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4574390Z 2025-12-04T09:44:20.4574501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4574887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4575231Z res = mod(**inputs) 2025-12-04T09:44:20.4575612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4576028Z outputs = self.model.decoder( 2025-12-04T09:44:20.4576437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4576849Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4577214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4577604Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4578008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4578402Z return func(*args, **kwargs) 2025-12-04T09:44:20.4578800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4579242Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4579671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4580063Z return func(*args, **kwargs) 2025-12-04T09:44:20.4580461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4580882Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4581026Z 2025-12-04T09:44:20.4581137Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4581527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4581873Z res = mod(**inputs) 2025-12-04T09:44:20.4582262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4582672Z outputs = self.model.decoder( 2025-12-04T09:44:20.4583080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4583500Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4583865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4584253Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4584658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4585085Z return func(*args, **kwargs) 2025-12-04T09:44:20.4585487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4585961Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4586428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4586829Z return func(*args, **kwargs) 2025-12-04T09:44:20.4587222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4587735Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4587890Z 2025-12-04T09:44:20.4587987Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4588500Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4588888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4589252Z res = mod(**inputs) 2025-12-04T09:44:20.4589640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4590054Z outputs = self.model.decoder( 2025-12-04T09:44:20.4590469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4590898Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4591272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4591655Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4592059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4592454Z return func(*args, **kwargs) 2025-12-04T09:44:20.4592839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4593254Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4593656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4594028Z return func(*args, **kwargs) 2025-12-04T09:44:20.4594400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4594816Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4595271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4595764Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4595949Z 2025-12-04T09:44:20.4596059Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4596445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4596792Z res = mod(**inputs) 2025-12-04T09:44:20.4597174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4597592Z outputs = self.model.decoder( 2025-12-04T09:44:20.4598000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4598415Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4598784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4599173Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4599584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4599975Z return func(*args, **kwargs) 2025-12-04T09:44:20.4600562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4601115Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4601558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4601988Z return func(*args, **kwargs) 2025-12-04T09:44:20.4602413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4602843Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4602992Z 2025-12-04T09:44:20.4603111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4603494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4603852Z res = mod(**inputs) 2025-12-04T09:44:20.4604246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4604682Z outputs = self.model.decoder( 2025-12-04T09:44:20.4605095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4605514Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4605890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4606273Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4606682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4607083Z return func(*args, **kwargs) 2025-12-04T09:44:20.4607481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4607950Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4608147Z 2025-12-04T09:44:20.4608261Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4608645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4608984Z res = mod(**inputs) 2025-12-04T09:44:20.4609371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4609790Z outputs = self.model.decoder( 2025-12-04T09:44:20.4610190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4610608Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4610983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4611373Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4611779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4612178Z return func(*args, **kwargs) 2025-12-04T09:44:20.4612583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4613045Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4613459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4613834Z return self.act(input) 2025-12-04T09:44:20.4613953Z 2025-12-04T09:44:20.4614073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4614450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4614796Z res = mod(**inputs) 2025-12-04T09:44:20.4615202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4615625Z outputs = self.model.decoder( 2025-12-04T09:44:20.4616027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4616472Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4616875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4617260Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4617670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4618073Z return func(*args, **kwargs) 2025-12-04T09:44:20.4618503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4618919Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4619100Z 2025-12-04T09:44:20.4619211Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4619600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4619945Z res = mod(**inputs) 2025-12-04T09:44:20.4620359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4620791Z outputs = self.model.decoder( 2025-12-04T09:44:20.4621215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4621642Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4622016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4622400Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4622807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4623202Z return func(*args, **kwargs) 2025-12-04T09:44:20.4623601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4624042Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4624459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4624857Z return func(*args, **kwargs) 2025-12-04T09:44:20.4625256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4625751Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4625970Z 2025-12-04T09:44:20.4626086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4626486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4626842Z res = mod(**inputs) 2025-12-04T09:44:20.4627240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4627664Z outputs = self.model.decoder( 2025-12-04T09:44:20.4628082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4628510Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4628888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4629296Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4629712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4630129Z return func(*args, **kwargs) 2025-12-04T09:44:20.4630556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4631012Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4631479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4631915Z return func(*args, **kwargs) 2025-12-04T09:44:20.4632319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4632752Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4632900Z 2025-12-04T09:44:20.4633024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4633410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4633762Z res = mod(**inputs) 2025-12-04T09:44:20.4634164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4634612Z outputs = self.model.decoder( 2025-12-04T09:44:20.4635023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4635455Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4635843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4636236Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4636657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4637067Z return func(*args, **kwargs) 2025-12-04T09:44:20.4637482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4637928Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4638369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4638783Z return func(*args, **kwargs) 2025-12-04T09:44:20.4639188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4639631Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4639793Z 2025-12-04T09:44:20.4639884Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4640154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4640643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4641010Z res = mod(**inputs) 2025-12-04T09:44:20.4641416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4641854Z outputs = self.model.decoder( 2025-12-04T09:44:20.4642268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4642704Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4643095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4643491Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4643912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4644328Z return func(*args, **kwargs) 2025-12-04T09:44:20.4644742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4645193Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4645671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4646075Z return func(*args, **kwargs) 2025-12-04T09:44:20.4646469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4646946Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4647656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4648198Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4648405Z 2025-12-04T09:44:20.4648524Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4648925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4649284Z res = mod(**inputs) 2025-12-04T09:44:20.4649689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4650148Z outputs = self.model.decoder( 2025-12-04T09:44:20.4650575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4651008Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4651388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4651790Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4652206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4652613Z return func(*args, **kwargs) 2025-12-04T09:44:20.4653019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4653471Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4653915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4654329Z return func(*args, **kwargs) 2025-12-04T09:44:20.4654725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4655152Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4655297Z 2025-12-04T09:44:20.4655416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4655798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4656145Z res = mod(**inputs) 2025-12-04T09:44:20.4656535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4656953Z outputs = self.model.decoder( 2025-12-04T09:44:20.4657357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4657777Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4658149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4658532Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4658942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4659341Z return func(*args, **kwargs) 2025-12-04T09:44:20.4659741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4660201Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4660397Z 2025-12-04T09:44:20.4660513Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4660939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4661311Z res = mod(**inputs) 2025-12-04T09:44:20.4661705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4662175Z outputs = self.model.decoder( 2025-12-04T09:44:20.4662603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4663012Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4663387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4663776Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4664184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4664575Z return func(*args, **kwargs) 2025-12-04T09:44:20.4664983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4665468Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4665880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4666253Z return self.act(input) 2025-12-04T09:44:20.4666381Z 2025-12-04T09:44:20.4666493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4666882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4667216Z res = mod(**inputs) 2025-12-04T09:44:20.4667602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4668018Z outputs = self.model.decoder( 2025-12-04T09:44:20.4668423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4668831Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4669206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4669594Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4669991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4670387Z return func(*args, **kwargs) 2025-12-04T09:44:20.4670785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4671205Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4671356Z 2025-12-04T09:44:20.4671467Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4671854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4672201Z res = mod(**inputs) 2025-12-04T09:44:20.4672581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4672999Z outputs = self.model.decoder( 2025-12-04T09:44:20.4673409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4673822Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4674188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4674578Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4674993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4675460Z return func(*args, **kwargs) 2025-12-04T09:44:20.4675910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:44:20.4676369Z hidden_states = residual + hidden_states 2025-12-04T09:44:20.4676523Z 2025-12-04T09:44:20.4676648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4677086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4677481Z res = mod(**inputs) 2025-12-04T09:44:20.4677882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4678327Z outputs = self.model.decoder( 2025-12-04T09:44:20.4678761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4679204Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4679595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4680016Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4680522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4680958Z return func(*args, **kwargs) 2025-12-04T09:44:20.4681381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4681841Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4682350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4682758Z return func(*args, **kwargs) 2025-12-04T09:44:20.4683157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4683658Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4683887Z 2025-12-04T09:44:20.4684002Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4684388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4684729Z res = mod(**inputs) 2025-12-04T09:44:20.4685117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4685533Z outputs = self.model.decoder( 2025-12-04T09:44:20.4685942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4686348Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4686723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4687114Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4687514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4687916Z return func(*args, **kwargs) 2025-12-04T09:44:20.4688316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4688760Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4689180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4689580Z return func(*args, **kwargs) 2025-12-04T09:44:20.4689981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4690404Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4690549Z 2025-12-04T09:44:20.4690662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4691052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4691421Z res = mod(**inputs) 2025-12-04T09:44:20.4691805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4692238Z outputs = self.model.decoder( 2025-12-04T09:44:20.4692670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4693092Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4693469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4693841Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4694243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4694639Z return func(*args, **kwargs) 2025-12-04T09:44:20.4695048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4695529Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4695962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4696360Z return func(*args, **kwargs) 2025-12-04T09:44:20.4696760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4697173Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4697317Z 2025-12-04T09:44:20.4697408Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4697651Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4698014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4698337Z res = mod(**inputs) 2025-12-04T09:44:20.4698696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4699108Z outputs = self.model.decoder( 2025-12-04T09:44:20.4699517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4699931Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4700299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4700686Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4701102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4701472Z return func(*args, **kwargs) 2025-12-04T09:44:20.4701852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4702277Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4702709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4703102Z return func(*args, **kwargs) 2025-12-04T09:44:20.4703502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4703947Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4704421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4704943Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4705146Z 2025-12-04T09:44:20.4705260Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4705646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4705986Z res = mod(**inputs) 2025-12-04T09:44:20.4706401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4706839Z outputs = self.model.decoder( 2025-12-04T09:44:20.4707246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4707677Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4708057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4708428Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4708805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4709184Z return func(*args, **kwargs) 2025-12-04T09:44:20.4709568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4710031Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4710453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4710854Z return func(*args, **kwargs) 2025-12-04T09:44:20.4711259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4711686Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4711835Z 2025-12-04T09:44:20.4711946Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4712332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4712675Z res = mod(**inputs) 2025-12-04T09:44:20.4713054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4713479Z outputs = self.model.decoder( 2025-12-04T09:44:20.4713889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4714307Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4714677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4715066Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4715473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4715866Z return func(*args, **kwargs) 2025-12-04T09:44:20.4716266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4716727Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4716912Z 2025-12-04T09:44:20.4717033Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4717412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4717757Z res = mod(**inputs) 2025-12-04T09:44:20.4718148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4718584Z outputs = self.model.decoder( 2025-12-04T09:44:20.4719007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4719438Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4719826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4720226Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4720755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4721212Z return func(*args, **kwargs) 2025-12-04T09:44:20.4721634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4722134Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4722618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4723008Z return self.act(input) 2025-12-04T09:44:20.4723135Z 2025-12-04T09:44:20.4723252Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4723668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4724019Z res = mod(**inputs) 2025-12-04T09:44:20.4724414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4724836Z outputs = self.model.decoder( 2025-12-04T09:44:20.4725301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4725747Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4726129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4726549Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4726979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4727386Z return func(*args, **kwargs) 2025-12-04T09:44:20.4727785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4728215Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4728376Z 2025-12-04T09:44:20.4728490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4728888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4729230Z res = mod(**inputs) 2025-12-04T09:44:20.4729626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4730050Z outputs = self.model.decoder( 2025-12-04T09:44:20.4730458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4730885Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4731268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4731664Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4732069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4732479Z return func(*args, **kwargs) 2025-12-04T09:44:20.4732894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4733342Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4733779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4734187Z return func(*args, **kwargs) 2025-12-04T09:44:20.4734599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4735098Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4735329Z 2025-12-04T09:44:20.4735444Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4735839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4736190Z res = mod(**inputs) 2025-12-04T09:44:20.4736610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4737060Z outputs = self.model.decoder( 2025-12-04T09:44:20.4737476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4737932Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4738325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4738727Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4739147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4739555Z return func(*args, **kwargs) 2025-12-04T09:44:20.4739972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4740455Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4740904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4741313Z return func(*args, **kwargs) 2025-12-04T09:44:20.4741728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4742171Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4742322Z 2025-12-04T09:44:20.4742437Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4742839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4743200Z res = mod(**inputs) 2025-12-04T09:44:20.4743603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4744032Z outputs = self.model.decoder( 2025-12-04T09:44:20.4744457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4744889Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4745271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4745673Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4746093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4746504Z return func(*args, **kwargs) 2025-12-04T09:44:20.4746917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4747663Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4748117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4748538Z return func(*args, **kwargs) 2025-12-04T09:44:20.4748951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4749402Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4749563Z 2025-12-04T09:44:20.4749666Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4749928Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4750331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4750696Z res = mod(**inputs) 2025-12-04T09:44:20.4751102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4751533Z outputs = self.model.decoder( 2025-12-04T09:44:20.4752018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4752451Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4752834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4753256Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4753700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4754106Z return func(*args, **kwargs) 2025-12-04T09:44:20.4754510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4754959Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4755394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4755800Z return func(*args, **kwargs) 2025-12-04T09:44:20.4756232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4756673Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4757160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4757674Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4757875Z 2025-12-04T09:44:20.4757988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4758370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4758721Z res = mod(**inputs) 2025-12-04T09:44:20.4759104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4759536Z outputs = self.model.decoder( 2025-12-04T09:44:20.4759962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4760435Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4760842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4761249Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4761670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4762071Z return func(*args, **kwargs) 2025-12-04T09:44:20.4762470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4762911Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4763337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4763733Z return func(*args, **kwargs) 2025-12-04T09:44:20.4764133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4764568Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4764719Z 2025-12-04T09:44:20.4764837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4765233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4765585Z res = mod(**inputs) 2025-12-04T09:44:20.4765980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4766408Z outputs = self.model.decoder( 2025-12-04T09:44:20.4766816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4767230Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4767627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4768039Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4768447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4768869Z return func(*args, **kwargs) 2025-12-04T09:44:20.4769268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4769730Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4769914Z 2025-12-04T09:44:20.4770037Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4770424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4770763Z res = mod(**inputs) 2025-12-04T09:44:20.4771148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4771591Z outputs = self.model.decoder( 2025-12-04T09:44:20.4771994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4772417Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4772796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4773189Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4773590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4773992Z return func(*args, **kwargs) 2025-12-04T09:44:20.4774392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4774853Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4775276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4775652Z return self.act(input) 2025-12-04T09:44:20.4775772Z 2025-12-04T09:44:20.4775891Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4776274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4776618Z res = mod(**inputs) 2025-12-04T09:44:20.4777005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4777472Z outputs = self.model.decoder( 2025-12-04T09:44:20.4777874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4778292Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4778674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4779063Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4779475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4779879Z return func(*args, **kwargs) 2025-12-04T09:44:20.4780290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4780710Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4780866Z 2025-12-04T09:44:20.4780977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4781363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4781700Z res = mod(**inputs) 2025-12-04T09:44:20.4782116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4782538Z outputs = self.model.decoder( 2025-12-04T09:44:20.4782949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4783378Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4783771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4784165Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4784575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4784985Z return func(*args, **kwargs) 2025-12-04T09:44:20.4785394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:44:20.4785820Z hidden_states = residual + hidden_states 2025-12-04T09:44:20.4785990Z 2025-12-04T09:44:20.4786103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4786487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4786831Z res = mod(**inputs) 2025-12-04T09:44:20.4787222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4787637Z outputs = self.model.decoder( 2025-12-04T09:44:20.4788049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4788467Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4788834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4789290Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4789703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4790107Z return func(*args, **kwargs) 2025-12-04T09:44:20.4790504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4790948Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4791398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4791803Z return func(*args, **kwargs) 2025-12-04T09:44:20.4792197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4792695Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4792913Z 2025-12-04T09:44:20.4793033Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4793413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4793790Z res = mod(**inputs) 2025-12-04T09:44:20.4794198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4794643Z outputs = self.model.decoder( 2025-12-04T09:44:20.4795070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4795482Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4795856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4796244Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4796675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4797092Z return func(*args, **kwargs) 2025-12-04T09:44:20.4797589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4798055Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4798522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4798960Z return func(*args, **kwargs) 2025-12-04T09:44:20.4799374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4799802Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4799957Z 2025-12-04T09:44:20.4800071Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4800553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4800920Z res = mod(**inputs) 2025-12-04T09:44:20.4801330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4801804Z outputs = self.model.decoder( 2025-12-04T09:44:20.4802225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4802659Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4803060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4803472Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4803939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4804361Z return func(*args, **kwargs) 2025-12-04T09:44:20.4804780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4805233Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4805677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4806098Z return func(*args, **kwargs) 2025-12-04T09:44:20.4806511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4806965Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4807119Z 2025-12-04T09:44:20.4807209Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4807478Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4807874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4808231Z res = mod(**inputs) 2025-12-04T09:44:20.4808631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4809041Z outputs = self.model.decoder( 2025-12-04T09:44:20.4809434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4809844Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4810217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4810608Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4811004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4811404Z return func(*args, **kwargs) 2025-12-04T09:44:20.4811804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4812240Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4812666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4813049Z return func(*args, **kwargs) 2025-12-04T09:44:20.4813445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4813917Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4814407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4814931Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4815137Z 2025-12-04T09:44:20.4815251Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4815617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4815950Z res = mod(**inputs) 2025-12-04T09:44:20.4816336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4816788Z outputs = self.model.decoder( 2025-12-04T09:44:20.4817199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4817623Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4818001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4818398Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4818810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4819213Z return func(*args, **kwargs) 2025-12-04T09:44:20.4819622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4820064Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4820498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4820900Z return func(*args, **kwargs) 2025-12-04T09:44:20.4821306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4821730Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4821886Z 2025-12-04T09:44:20.4821997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4822385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4822720Z res = mod(**inputs) 2025-12-04T09:44:20.4823112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4823533Z outputs = self.model.decoder( 2025-12-04T09:44:20.4823942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4824357Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4824736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4825130Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4825534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4825937Z return func(*args, **kwargs) 2025-12-04T09:44:20.4826344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4826811Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4826999Z 2025-12-04T09:44:20.4827112Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4827533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4827883Z res = mod(**inputs) 2025-12-04T09:44:20.4828273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4828710Z outputs = self.model.decoder( 2025-12-04T09:44:20.4829142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4829567Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4829939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4830330Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4830738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4831140Z return func(*args, **kwargs) 2025-12-04T09:44:20.4831539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4832032Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4832453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4832814Z return self.act(input) 2025-12-04T09:44:20.4832943Z 2025-12-04T09:44:20.4833056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4833443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4833789Z res = mod(**inputs) 2025-12-04T09:44:20.4834166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4834604Z outputs = self.model.decoder( 2025-12-04T09:44:20.4835033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4835449Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4835827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4836234Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4836653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4837058Z return func(*args, **kwargs) 2025-12-04T09:44:20.4837473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4837907Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4838058Z 2025-12-04T09:44:20.4838180Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4838565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4838925Z res = mod(**inputs) 2025-12-04T09:44:20.4839323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4839750Z outputs = self.model.decoder( 2025-12-04T09:44:20.4840182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4840716Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4841112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4841516Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4841949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4842373Z return func(*args, **kwargs) 2025-12-04T09:44:20.4842868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4843328Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4843788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4844229Z return func(*args, **kwargs) 2025-12-04T09:44:20.4844665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4845186Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4845427Z 2025-12-04T09:44:20.4845545Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4845949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4846305Z res = mod(**inputs) 2025-12-04T09:44:20.4846711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4847310Z outputs = self.model.decoder( 2025-12-04T09:44:20.4847735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4848236Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4848638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4849046Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4849466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4849885Z return func(*args, **kwargs) 2025-12-04T09:44:20.4850309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4850969Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4851660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4852195Z return func(*args, **kwargs) 2025-12-04T09:44:20.4852614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4853059Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4853221Z 2025-12-04T09:44:20.4853338Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4853739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4854100Z res = mod(**inputs) 2025-12-04T09:44:20.4854495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4854927Z outputs = self.model.decoder( 2025-12-04T09:44:20.4855351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4855773Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4856163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4856569Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4856978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4857368Z return func(*args, **kwargs) 2025-12-04T09:44:20.4857769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4858210Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4858636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4859032Z return func(*args, **kwargs) 2025-12-04T09:44:20.4859512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4859957Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4860143Z 2025-12-04T09:44:20.4860232Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4860498Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4860924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4861288Z res = mod(**inputs) 2025-12-04T09:44:20.4861684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4862117Z outputs = self.model.decoder( 2025-12-04T09:44:20.4862539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4862961Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4863390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4863793Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4864216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4864624Z return func(*args, **kwargs) 2025-12-04T09:44:20.4865040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4865501Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4865944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4866350Z return func(*args, **kwargs) 2025-12-04T09:44:20.4866765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4867230Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4867725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4868269Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4868274Z 2025-12-04T09:44:20.4868399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4868622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4868693Z res = mod(**inputs) 2025-12-04T09:44:20.4868991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4869073Z outputs = self.model.decoder( 2025-12-04T09:44:20.4869354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4869454Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4869695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4869790Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4870051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4870126Z return func(*args, **kwargs) 2025-12-04T09:44:20.4870406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4870509Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4870779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4870852Z return func(*args, **kwargs) 2025-12-04T09:44:20.4871162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4871263Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4871285Z 2025-12-04T09:44:20.4871396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4871623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4871703Z res = mod(**inputs) 2025-12-04T09:44:20.4871974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4872059Z outputs = self.model.decoder( 2025-12-04T09:44:20.4872324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4872398Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4872633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4872731Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4872975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4873053Z return func(*args, **kwargs) 2025-12-04T09:44:20.4873308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4873439Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4873442Z 2025-12-04T09:44:20.4873548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4873748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4873821Z res = mod(**inputs) 2025-12-04T09:44:20.4874078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4874162Z outputs = self.model.decoder( 2025-12-04T09:44:20.4874422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4874497Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4874731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4874810Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4875053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4875129Z return func(*args, **kwargs) 2025-12-04T09:44:20.4875385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4875514Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4875733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4875806Z return self.act(input) 2025-12-04T09:44:20.4875809Z 2025-12-04T09:44:20.4875923Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4876124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4876196Z res = mod(**inputs) 2025-12-04T09:44:20.4876462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4876540Z outputs = self.model.decoder( 2025-12-04T09:44:20.4876815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4876891Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4877126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4877238Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4877494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4877596Z return func(*args, **kwargs) 2025-12-04T09:44:20.4877894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4877983Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4877987Z 2025-12-04T09:44:20.4878108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4878318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4878387Z res = mod(**inputs) 2025-12-04T09:44:20.4878663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4878739Z outputs = self.model.decoder( 2025-12-04T09:44:20.4879038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4879116Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4879351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4879443Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4879702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4879783Z return func(*args, **kwargs) 2025-12-04T09:44:20.4880052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:44:20.4880139Z hidden_states = residual + hidden_states 2025-12-04T09:44:20.4880143Z 2025-12-04T09:44:20.4880262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4880633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4880714Z res = mod(**inputs) 2025-12-04T09:44:20.4881015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4881099Z outputs = self.model.decoder( 2025-12-04T09:44:20.4881406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4881485Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4881727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4881820Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4882090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4882165Z return func(*args, **kwargs) 2025-12-04T09:44:20.4882454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4882565Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4882843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4882917Z return func(*args, **kwargs) 2025-12-04T09:44:20.4883176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4883341Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4883345Z 2025-12-04T09:44:20.4883451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4883660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4883728Z res = mod(**inputs) 2025-12-04T09:44:20.4884019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4884127Z outputs = self.model.decoder( 2025-12-04T09:44:20.4884382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4884470Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4884710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4884794Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4885059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4885133Z return func(*args, **kwargs) 2025-12-04T09:44:20.4885410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4885538Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4885782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4885858Z return func(*args, **kwargs) 2025-12-04T09:44:20.4886117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4886209Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4886213Z 2025-12-04T09:44:20.4886324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4886523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4886588Z res = mod(**inputs) 2025-12-04T09:44:20.4886853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4886927Z outputs = self.model.decoder( 2025-12-04T09:44:20.4887192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4887268Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4887491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4887579Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4887822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4887899Z return func(*args, **kwargs) 2025-12-04T09:44:20.4888155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4888254Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4888508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4888579Z return func(*args, **kwargs) 2025-12-04T09:44:20.4888834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4888932Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4888936Z 2025-12-04T09:44:20.4889019Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4889132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4889331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4889397Z res = mod(**inputs) 2025-12-04T09:44:20.4889662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4889735Z outputs = self.model.decoder( 2025-12-04T09:44:20.4890012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4890095Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4890318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4890424Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4890682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4890753Z return func(*args, **kwargs) 2025-12-04T09:44:20.4891028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4891131Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4891393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4891464Z return func(*args, **kwargs) 2025-12-04T09:44:20.4891764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4891878Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4892211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4892413Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4892426Z 2025-12-04T09:44:20.4892577Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4892835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4892911Z res = mod(**inputs) 2025-12-04T09:44:20.4893180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4893265Z outputs = self.model.decoder( 2025-12-04T09:44:20.4893530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4893604Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4893838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4893927Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4894186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4894266Z return func(*args, **kwargs) 2025-12-04T09:44:20.4894532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4894635Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4894899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4894975Z return func(*args, **kwargs) 2025-12-04T09:44:20.4895252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4895340Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4895344Z 2025-12-04T09:44:20.4895453Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4895669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4895737Z res = mod(**inputs) 2025-12-04T09:44:20.4896007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4896092Z outputs = self.model.decoder( 2025-12-04T09:44:20.4896361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4896444Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4896706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4896815Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4897082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4897173Z return func(*args, **kwargs) 2025-12-04T09:44:20.4897454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4897581Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4897585Z 2025-12-04T09:44:20.4897695Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4897916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4897986Z res = mod(**inputs) 2025-12-04T09:44:20.4898259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4898372Z outputs = self.model.decoder( 2025-12-04T09:44:20.4898646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4898735Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4898973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4899083Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4899350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4899424Z return func(*args, **kwargs) 2025-12-04T09:44:20.4899700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4899828Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4900060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4900145Z return self.act(input) 2025-12-04T09:44:20.4900149Z 2025-12-04T09:44:20.4900258Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4900472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4900553Z res = mod(**inputs) 2025-12-04T09:44:20.4900825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4900910Z outputs = self.model.decoder( 2025-12-04T09:44:20.4901178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4901255Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4901497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4901582Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4901844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4901928Z return func(*args, **kwargs) 2025-12-04T09:44:20.4902198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4902292Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4902296Z 2025-12-04T09:44:20.4902404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4902615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4902694Z res = mod(**inputs) 2025-12-04T09:44:20.4902991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4903078Z outputs = self.model.decoder( 2025-12-04T09:44:20.4903353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4903453Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4903712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4903799Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4904058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4904140Z return func(*args, **kwargs) 2025-12-04T09:44:20.4904412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4904531Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4905698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4905772Z return func(*args, **kwargs) 2025-12-04T09:44:20.4906055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4906220Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4906224Z 2025-12-04T09:44:20.4906343Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4906554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4906622Z res = mod(**inputs) 2025-12-04T09:44:20.4906900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4906979Z outputs = self.model.decoder( 2025-12-04T09:44:20.4907255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4907343Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4907581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4907685Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4907931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4907999Z return func(*args, **kwargs) 2025-12-04T09:44:20.4908260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4908360Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4908603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4908682Z return func(*args, **kwargs) 2025-12-04T09:44:20.4908935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4909025Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4909028Z 2025-12-04T09:44:20.4909134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4909337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4909412Z res = mod(**inputs) 2025-12-04T09:44:20.4909667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4909746Z outputs = self.model.decoder( 2025-12-04T09:44:20.4910001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4910074Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4910325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4910406Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4910672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4910769Z return func(*args, **kwargs) 2025-12-04T09:44:20.4911040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4911148Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4911406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4911481Z return func(*args, **kwargs) 2025-12-04T09:44:20.4911760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4911883Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4911887Z 2025-12-04T09:44:20.4911976Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4912082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4912284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4912357Z res = mod(**inputs) 2025-12-04T09:44:20.4912617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4912689Z outputs = self.model.decoder( 2025-12-04T09:44:20.4912955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4913028Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4913270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4913355Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4913615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4913697Z return func(*args, **kwargs) 2025-12-04T09:44:20.4913970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4914075Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4914343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4914415Z return func(*args, **kwargs) 2025-12-04T09:44:20.4914694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4914801Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4915127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4915273Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4915277Z 2025-12-04T09:44:20.4915384Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4915596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4915666Z res = mod(**inputs) 2025-12-04T09:44:20.4915943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4916029Z outputs = self.model.decoder( 2025-12-04T09:44:20.4916302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4916378Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4916646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4916737Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4917022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4917097Z return func(*args, **kwargs) 2025-12-04T09:44:20.4917397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4917510Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4917772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4917851Z return func(*args, **kwargs) 2025-12-04T09:44:20.4918119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4918225Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4918229Z 2025-12-04T09:44:20.4918346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4918563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4918631Z res = mod(**inputs) 2025-12-04T09:44:20.4918913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4918991Z outputs = self.model.decoder( 2025-12-04T09:44:20.4919268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4919345Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4919581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4919670Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4919933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4920004Z return func(*args, **kwargs) 2025-12-04T09:44:20.4920283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4920534Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4920543Z 2025-12-04T09:44:20.4920681Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4920901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4920973Z res = mod(**inputs) 2025-12-04T09:44:20.4921267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4921346Z outputs = self.model.decoder( 2025-12-04T09:44:20.4921658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4921741Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4921989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4922087Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4922361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4922447Z return func(*args, **kwargs) 2025-12-04T09:44:20.4922730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4922857Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4923098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4923175Z return self.act(input) 2025-12-04T09:44:20.4923208Z 2025-12-04T09:44:20.4923321Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4923542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4923629Z res = mod(**inputs) 2025-12-04T09:44:20.4923922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4924000Z outputs = self.model.decoder( 2025-12-04T09:44:20.4924269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4924353Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4924586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4924668Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4924936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4925029Z return func(*args, **kwargs) 2025-12-04T09:44:20.4925302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4925393Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4925399Z 2025-12-04T09:44:20.4925509Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4925726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4925794Z res = mod(**inputs) 2025-12-04T09:44:20.4926065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4926148Z outputs = self.model.decoder( 2025-12-04T09:44:20.4926417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4926499Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4926733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4926817Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4927085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4927158Z return func(*args, **kwargs) 2025-12-04T09:44:20.4927434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:44:20.4927518Z hidden_states = residual + hidden_states 2025-12-04T09:44:20.4927522Z 2025-12-04T09:44:20.4927632Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4927849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4927919Z res = mod(**inputs) 2025-12-04T09:44:20.4928192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4928278Z outputs = self.model.decoder( 2025-12-04T09:44:20.4928549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4928632Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4928866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4928949Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4929214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4929288Z return func(*args, **kwargs) 2025-12-04T09:44:20.4929583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4929698Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4929957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4930065Z return func(*args, **kwargs) 2025-12-04T09:44:20.4930361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4930526Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4930530Z 2025-12-04T09:44:20.4930649Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4930861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4930938Z res = mod(**inputs) 2025-12-04T09:44:20.4931218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4931317Z outputs = self.model.decoder( 2025-12-04T09:44:20.4931611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4931691Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4931939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4932032Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4932301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4932383Z return func(*args, **kwargs) 2025-12-04T09:44:20.4932669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4932778Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4933078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4933176Z return func(*args, **kwargs) 2025-12-04T09:44:20.4933589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4933678Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4933682Z 2025-12-04T09:44:20.4933791Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4934008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4934075Z res = mod(**inputs) 2025-12-04T09:44:20.4934346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4934431Z outputs = self.model.decoder( 2025-12-04T09:44:20.4934702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4934788Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4935023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4935111Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4935378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4935450Z return func(*args, **kwargs) 2025-12-04T09:44:20.4935720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4935831Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4936090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4936171Z return func(*args, **kwargs) 2025-12-04T09:44:20.4936462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4936557Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4936578Z 2025-12-04T09:44:20.4936675Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4936786Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4937020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4937090Z res = mod(**inputs) 2025-12-04T09:44:20.4937363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4937449Z outputs = self.model.decoder( 2025-12-04T09:44:20.4937719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4937796Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4938061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4938145Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4938412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4938487Z return func(*args, **kwargs) 2025-12-04T09:44:20.4938753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4938865Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4939125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4939208Z return func(*args, **kwargs) 2025-12-04T09:44:20.4939479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4939586Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4939907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4940052Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4940056Z 2025-12-04T09:44:20.4940168Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4940389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4940460Z res = mod(**inputs) 2025-12-04T09:44:20.4940740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4940816Z outputs = self.model.decoder( 2025-12-04T09:44:20.4941085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4941173Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4941412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4941497Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4941763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4941836Z return func(*args, **kwargs) 2025-12-04T09:44:20.4942112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4942216Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4942476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4942555Z return func(*args, **kwargs) 2025-12-04T09:44:20.4942847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4942942Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4942974Z 2025-12-04T09:44:20.4943084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4943310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4943392Z res = mod(**inputs) 2025-12-04T09:44:20.4943665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4943740Z outputs = self.model.decoder( 2025-12-04T09:44:20.4944017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4944094Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4944338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4944438Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4944694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4944777Z return func(*args, **kwargs) 2025-12-04T09:44:20.4945045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4945178Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4945182Z 2025-12-04T09:44:20.4945289Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4945500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4945577Z res = mod(**inputs) 2025-12-04T09:44:20.4945850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4945934Z outputs = self.model.decoder( 2025-12-04T09:44:20.4946216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4946294Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4946541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4946623Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4946881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4946963Z return func(*args, **kwargs) 2025-12-04T09:44:20.4947419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4947551Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4947792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4947870Z return self.act(input) 2025-12-04T09:44:20.4947874Z 2025-12-04T09:44:20.4947994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4948207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4948277Z res = mod(**inputs) 2025-12-04T09:44:20.4948557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4948633Z outputs = self.model.decoder( 2025-12-04T09:44:20.4948912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4948989Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4949231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4949371Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4949630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4949732Z return func(*args, **kwargs) 2025-12-04T09:44:20.4950032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4950121Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4950125Z 2025-12-04T09:44:20.4950241Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4950452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4950540Z res = mod(**inputs) 2025-12-04T09:44:20.4950833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4950912Z outputs = self.model.decoder( 2025-12-04T09:44:20.4951224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4951303Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4951542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4951635Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4951892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4951965Z return func(*args, **kwargs) 2025-12-04T09:44:20.4952249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4952352Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4952614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4952691Z return func(*args, **kwargs) 2025-12-04T09:44:20.4952959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:44:20.4953129Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:20.4953133Z 2025-12-04T09:44:20.4953246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4953465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4953534Z res = mod(**inputs) 2025-12-04T09:44:20.4953806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4953890Z outputs = self.model.decoder( 2025-12-04T09:44:20.4954164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4954243Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4954489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4954573Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4954839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4954910Z return func(*args, **kwargs) 2025-12-04T09:44:20.4955178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4955290Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4955547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4955620Z return func(*args, **kwargs) 2025-12-04T09:44:20.4955921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:44:20.4956010Z key_states = self.k_proj(current_states) 2025-12-04T09:44:20.4956033Z 2025-12-04T09:44:20.4956150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4956363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4956450Z res = mod(**inputs) 2025-12-04T09:44:20.4956728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4956804Z outputs = self.model.decoder( 2025-12-04T09:44:20.4957079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4957155Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4957390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4957536Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4957793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4957869Z return func(*args, **kwargs) 2025-12-04T09:44:20.4958149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4958255Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4958521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4958595Z return func(*args, **kwargs) 2025-12-04T09:44:20.4958863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:44:20.4958965Z value_states = self.v_proj(current_states) 2025-12-04T09:44:20.4958969Z 2025-12-04T09:44:20.4959060Z cudagraph partition due to non gpu ops 2025-12-04T09:44:20.4959171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4959392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4959464Z res = mod(**inputs) 2025-12-04T09:44:20.4959744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4959825Z outputs = self.model.decoder( 2025-12-04T09:44:20.4960097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4960185Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4960495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4960595Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4960863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4960941Z return func(*args, **kwargs) 2025-12-04T09:44:20.4961227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4961337Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4961603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4961686Z return func(*args, **kwargs) 2025-12-04T09:44:20.4961974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:44:20.4962086Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:20.4962399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:20.4962570Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:20.4962575Z 2025-12-04T09:44:20.4962696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4962927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4963023Z res = mod(**inputs) 2025-12-04T09:44:20.4963305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4963384Z outputs = self.model.decoder( 2025-12-04T09:44:20.4963667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4963744Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4963985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4964077Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4964355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4964438Z return func(*args, **kwargs) 2025-12-04T09:44:20.4964711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:44:20.4964814Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:20.4965080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4965154Z return func(*args, **kwargs) 2025-12-04T09:44:20.4965426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:44:20.4965522Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:20.4965525Z 2025-12-04T09:44:20.4965638Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4965860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4965930Z res = mod(**inputs) 2025-12-04T09:44:20.4966204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4966292Z outputs = self.model.decoder( 2025-12-04T09:44:20.4966570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4966649Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4966877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4966955Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4967216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4967294Z return func(*args, **kwargs) 2025-12-04T09:44:20.4967566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4967700Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4967704Z 2025-12-04T09:44:20.4967816Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4968035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4968104Z res = mod(**inputs) 2025-12-04T09:44:20.4968381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4968466Z outputs = self.model.decoder( 2025-12-04T09:44:20.4968739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4968853Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4969096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4969200Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4969465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4969555Z return func(*args, **kwargs) 2025-12-04T09:44:20.4969826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:44:20.4969957Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:20.4970186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:20.4970268Z return self.act(input) 2025-12-04T09:44:20.4970272Z 2025-12-04T09:44:20.4970381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4970614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4970695Z res = mod(**inputs) 2025-12-04T09:44:20.4970965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4971040Z outputs = self.model.decoder( 2025-12-04T09:44:20.4971320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4971397Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4971643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4971726Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4971985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4972068Z return func(*args, **kwargs) 2025-12-04T09:44:20.4972340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:44:20.4972435Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:20.4972439Z 2025-12-04T09:44:20.4972548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4972763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4972836Z res = mod(**inputs) 2025-12-04T09:44:20.4973092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:44:20.4973164Z outputs = self.model.decoder( 2025-12-04T09:44:20.4973424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:44:20.4973496Z layer_outputs = decoder_layer( 2025-12-04T09:44:20.4973726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:20.4973805Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:20.4974137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:20.4974240Z return func(*args, **kwargs) 2025-12-04T09:44:20.4974609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:44:20.4974697Z hidden_states = residual + hidden_states 2025-12-04T09:44:20.4974702Z 2025-12-04T09:44:20.4974805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4975004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4975077Z res = mod(**inputs) 2025-12-04T09:44:20.4975352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1881, in forward 2025-12-04T09:44:20.4975437Z logits = self.lm_head(outputs[0]) 2025-12-04T09:44:20.4975441Z 2025-12-04T09:44:20.4975552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:20.4975771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:20.4975868Z res = mod(**inputs) 2025-12-04T09:44:20.4976142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1887, in forward 2025-12-04T09:44:20.4976301Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:44:20.4976305Z 2025-12-04T09:44:31.6484721Z Compilation time (from dynamo_timed): 17.277774514 2025-12-04T09:44:31.6689442Z pass 2025-12-04T09:44:31.6694952Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:31.6696362Z TIMING: _recursive_pre_grad_passes:0.0066 _recursive_joint_graph_passes:0.69186 _recursive_post_grad_passes:0.07199 async_compile.wait:0.91031 code_gen:10.28831 inductor_compile:11.68984 backend_compile:14.92612 gc:0.00036 entire_frame_compile:17.27777 total_wall_time:17.27777 2025-12-04T09:44:31.6697829Z STATS: call_* op count: 337 | FakeTensorMode.__torch_dispatch__:7389 | FakeTensor.__torch_dispatch__:4449 | ProxyTorchDispatchMode.__torch_dispatch__:2049 2025-12-04T09:44:31.6698440Z Dynamo produced 1 graphs covering 337 ops with 0 graph breaks (0 unique) 2025-12-04T09:44:34.2200723Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:44:34.2201682Z import pynvml # type: ignore[import] 2025-12-04T09:44:37.6295158Z 2025-12-04T09:44:40.1431599Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:44:40.1431956Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:44:40.1445808Z cpu eval MT5ForConditionalGeneration 2025-12-04T09:44:40.7637817Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:41.0297956Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:41.2927203Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:52.7345846Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7346353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7346722Z res = mod(**inputs) 2025-12-04T09:44:52.7347324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7347773Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7348233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7348693Z layer_outputs = layer_module( 2025-12-04T09:44:52.7349130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7349545Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7349970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7350380Z return func(*args, **kwargs) 2025-12-04T09:44:52.7350796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7351219Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7351630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7352343Z return func(*args, **kwargs) 2025-12-04T09:44:52.7352737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7353263Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7353731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7354137Z return func(*args, **kwargs) 2025-12-04T09:44:52.7354527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 424, in forward 2025-12-04T09:44:52.7354950Z position_bias = position_bias + causal_mask 2025-12-04T09:44:52.7355115Z 2025-12-04T09:44:52.7355232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7355633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7355986Z res = mod(**inputs) 2025-12-04T09:44:52.7356375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7356869Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7357305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7357726Z layer_outputs = layer_module( 2025-12-04T09:44:52.7358118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7358526Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7359046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7359475Z return func(*args, **kwargs) 2025-12-04T09:44:52.7359882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7360530Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7360974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7361376Z return func(*args, **kwargs) 2025-12-04T09:44:52.7361778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7362215Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7362623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7363023Z return func(*args, **kwargs) 2025-12-04T09:44:52.7363445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7363854Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7364007Z 2025-12-04T09:44:52.7364127Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7364538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7364898Z res = mod(**inputs) 2025-12-04T09:44:52.7365899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7368826Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7369316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7369751Z layer_outputs = layer_module( 2025-12-04T09:44:52.7370150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7370567Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7371039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7371767Z return func(*args, **kwargs) 2025-12-04T09:44:52.7372236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7372886Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7373319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7373764Z return func(*args, **kwargs) 2025-12-04T09:44:52.7374177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7374609Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7375048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7375451Z return func(*args, **kwargs) 2025-12-04T09:44:52.7375952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7376573Z key_states = self.k(current_states) 2025-12-04T09:44:52.7376734Z 2025-12-04T09:44:52.7376865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7377359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7377807Z res = mod(**inputs) 2025-12-04T09:44:52.7378219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7378720Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7379152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7379708Z layer_outputs = layer_module( 2025-12-04T09:44:52.7380249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7380837Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7381448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7382026Z return func(*args, **kwargs) 2025-12-04T09:44:52.7382446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7382848Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7383280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7383685Z return func(*args, **kwargs) 2025-12-04T09:44:52.7384084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7384510Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7384932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7385388Z return func(*args, **kwargs) 2025-12-04T09:44:52.7385793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7386229Z value_states = self.v(current_states) 2025-12-04T09:44:52.7386447Z 2025-12-04T09:44:52.7386548Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7386797Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7387153Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7387562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7387917Z res = mod(**inputs) 2025-12-04T09:44:52.7388315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7388728Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7389160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7389572Z layer_outputs = layer_module( 2025-12-04T09:44:52.7389980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7390380Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7390827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7391340Z return func(*args, **kwargs) 2025-12-04T09:44:52.7391914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7392433Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7392838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7393244Z return func(*args, **kwargs) 2025-12-04T09:44:52.7393684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7394263Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7394852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7395425Z return func(*args, **kwargs) 2025-12-04T09:44:52.7396048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7396554Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7396767Z 2025-12-04T09:44:52.7396924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7397506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7397982Z res = mod(**inputs) 2025-12-04T09:44:52.7398524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7399208Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7399728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7400298Z layer_outputs = layer_module( 2025-12-04T09:44:52.7400885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7401423Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7402028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7402628Z return func(*args, **kwargs) 2025-12-04T09:44:52.7403071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7403581Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7404018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7404474Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7404935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.7405388Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.7405569Z 2025-12-04T09:44:52.7405693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7406162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7406511Z res = mod(**inputs) 2025-12-04T09:44:52.7406895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7407315Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7407759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7408170Z layer_outputs = layer_module( 2025-12-04T09:44:52.7408570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7408999Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7409416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7409817Z return func(*args, **kwargs) 2025-12-04T09:44:52.7410291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7410714Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7411142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7411623Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7412060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.7412472Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.7412616Z 2025-12-04T09:44:52.7412738Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7413125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7413464Z res = mod(**inputs) 2025-12-04T09:44:52.7413841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7414248Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7414643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7415154Z layer_outputs = layer_module( 2025-12-04T09:44:52.7415540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7415945Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7416363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7416775Z return func(*args, **kwargs) 2025-12-04T09:44:52.7417169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7417585Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7418046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7418563Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7419012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.7419430Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.7419592Z 2025-12-04T09:44:52.7419705Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7420106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7420472Z res = mod(**inputs) 2025-12-04T09:44:52.7420845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7421324Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7421708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7422116Z layer_outputs = layer_module( 2025-12-04T09:44:52.7422512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7422932Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7423355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7423866Z return func(*args, **kwargs) 2025-12-04T09:44:52.7424274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7424698Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7425092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7425517Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7425951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.7426402Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.7426549Z 2025-12-04T09:44:52.7426665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7427078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7427491Z res = mod(**inputs) 2025-12-04T09:44:52.7427880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7428342Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7428732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7429122Z layer_outputs = layer_module( 2025-12-04T09:44:52.7429466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7429835Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7430236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7430639Z return func(*args, **kwargs) 2025-12-04T09:44:52.7431021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7431543Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7431959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7432359Z return func(*args, **kwargs) 2025-12-04T09:44:52.7432741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7433164Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7433577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7433969Z return func(*args, **kwargs) 2025-12-04T09:44:52.7434359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7434789Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7434933Z 2025-12-04T09:44:52.7435056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7435441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7435789Z res = mod(**inputs) 2025-12-04T09:44:52.7436161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7436563Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7436977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7437390Z layer_outputs = layer_module( 2025-12-04T09:44:52.7437780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7438196Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7438617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7439045Z return func(*args, **kwargs) 2025-12-04T09:44:52.7439459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7439884Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7440306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7440848Z return func(*args, **kwargs) 2025-12-04T09:44:52.7441248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7441687Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7442123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7442556Z return func(*args, **kwargs) 2025-12-04T09:44:52.7442918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7443307Z key_states = self.k(current_states) 2025-12-04T09:44:52.7443445Z 2025-12-04T09:44:52.7443564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7443950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7444298Z res = mod(**inputs) 2025-12-04T09:44:52.7444670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7445073Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7445465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7445861Z layer_outputs = layer_module( 2025-12-04T09:44:52.7446224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7446605Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7447013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7447552Z return func(*args, **kwargs) 2025-12-04T09:44:52.7447949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7448358Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7448770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7449166Z return func(*args, **kwargs) 2025-12-04T09:44:52.7449558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7449967Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7450377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7450775Z return func(*args, **kwargs) 2025-12-04T09:44:52.7451157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7451562Z value_states = self.v(current_states) 2025-12-04T09:44:52.7451713Z 2025-12-04T09:44:52.7451800Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7452029Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7452276Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7452657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7452995Z res = mod(**inputs) 2025-12-04T09:44:52.7453460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7453876Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7454289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7454679Z layer_outputs = layer_module( 2025-12-04T09:44:52.7455014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7455364Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7455726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7456079Z return func(*args, **kwargs) 2025-12-04T09:44:52.7456430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7456829Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7457202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7457555Z return func(*args, **kwargs) 2025-12-04T09:44:52.7457909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7458280Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7458643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7459002Z return func(*args, **kwargs) 2025-12-04T09:44:52.7459353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7459719Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7459846Z 2025-12-04T09:44:52.7459947Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7460299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7460609Z res = mod(**inputs) 2025-12-04T09:44:52.7460949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7461304Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7461668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7462048Z layer_outputs = layer_module( 2025-12-04T09:44:52.7462396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7462750Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7463120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7463482Z return func(*args, **kwargs) 2025-12-04T09:44:52.7463827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7464211Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7464594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7464993Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7465396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.7465785Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.7465934Z 2025-12-04T09:44:52.7466044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7466396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7466724Z res = mod(**inputs) 2025-12-04T09:44:52.7467083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7467482Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7467836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7468226Z layer_outputs = layer_module( 2025-12-04T09:44:52.7468569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7468925Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7469299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7469674Z return func(*args, **kwargs) 2025-12-04T09:44:52.7470020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7470417Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7470803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7471214Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7471614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.7471990Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.7472129Z 2025-12-04T09:44:52.7472231Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7472590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7472899Z res = mod(**inputs) 2025-12-04T09:44:52.7473249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7473628Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7473997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7474359Z layer_outputs = layer_module( 2025-12-04T09:44:52.7474706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7475066Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7475434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7475800Z return func(*args, **kwargs) 2025-12-04T09:44:52.7476157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7476543Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7476923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7477336Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7477737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.7478119Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.7478273Z 2025-12-04T09:44:52.7478381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7478744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7479069Z res = mod(**inputs) 2025-12-04T09:44:52.7479419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7479803Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7480199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7480711Z layer_outputs = layer_module( 2025-12-04T09:44:52.7481108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7481541Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7481995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7482386Z return func(*args, **kwargs) 2025-12-04T09:44:52.7482750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7483141Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7483528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7483971Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7484423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.7484850Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.7484997Z 2025-12-04T09:44:52.7485117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7485499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7485843Z res = mod(**inputs) 2025-12-04T09:44:52.7486224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7486601Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7486966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7487336Z layer_outputs = layer_module( 2025-12-04T09:44:52.7487710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7488093Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7488500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7488901Z return func(*args, **kwargs) 2025-12-04T09:44:52.7489283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7489696Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7490107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7490516Z return func(*args, **kwargs) 2025-12-04T09:44:52.7490897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7491309Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7491727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7492115Z return func(*args, **kwargs) 2025-12-04T09:44:52.7492514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7492988Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7493132Z 2025-12-04T09:44:52.7493254Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7493645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7493998Z res = mod(**inputs) 2025-12-04T09:44:52.7494385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7494803Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7495227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7495635Z layer_outputs = layer_module( 2025-12-04T09:44:52.7496005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7496417Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7496851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7497262Z return func(*args, **kwargs) 2025-12-04T09:44:52.7497658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7498076Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7498496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7498880Z return func(*args, **kwargs) 2025-12-04T09:44:52.7499244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7499659Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7500054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7500432Z return func(*args, **kwargs) 2025-12-04T09:44:52.7500793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7501181Z key_states = self.k(current_states) 2025-12-04T09:44:52.7501316Z 2025-12-04T09:44:52.7501431Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7501799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7502118Z res = mod(**inputs) 2025-12-04T09:44:52.7502479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7502865Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7503236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7503620Z layer_outputs = layer_module( 2025-12-04T09:44:52.7503977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7504345Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7504726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7505103Z return func(*args, **kwargs) 2025-12-04T09:44:52.7505484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7505867Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7506257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7506637Z return func(*args, **kwargs) 2025-12-04T09:44:52.7507008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7507397Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7507783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7508159Z return func(*args, **kwargs) 2025-12-04T09:44:52.7508521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7508909Z value_states = self.v(current_states) 2025-12-04T09:44:52.7509049Z 2025-12-04T09:44:52.7509132Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7509352Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7509609Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7509984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7510341Z res = mod(**inputs) 2025-12-04T09:44:52.7510691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7511095Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7511476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7511857Z layer_outputs = layer_module( 2025-12-04T09:44:52.7512204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7512572Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7512955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7513351Z return func(*args, **kwargs) 2025-12-04T09:44:52.7513711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7514099Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7514490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7514856Z return func(*args, **kwargs) 2025-12-04T09:44:52.7515224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7515612Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7516001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7516370Z return func(*args, **kwargs) 2025-12-04T09:44:52.7516736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7517122Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7517258Z 2025-12-04T09:44:52.7517368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7517738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7518071Z res = mod(**inputs) 2025-12-04T09:44:52.7518450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7518847Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7519248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7519650Z layer_outputs = layer_module( 2025-12-04T09:44:52.7520018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7520499Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7520927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7521345Z return func(*args, **kwargs) 2025-12-04T09:44:52.7521743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7522199Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7522605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7523040Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7523457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.7523870Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.7524028Z 2025-12-04T09:44:52.7524183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7524549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7524898Z res = mod(**inputs) 2025-12-04T09:44:52.7525272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7525655Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7526026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7526407Z layer_outputs = layer_module( 2025-12-04T09:44:52.7526763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7527126Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7527513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7527916Z return func(*args, **kwargs) 2025-12-04T09:44:52.7528289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7528687Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7529089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7529518Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7529938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.7530348Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.7530502Z 2025-12-04T09:44:52.7530613Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7530996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7531318Z res = mod(**inputs) 2025-12-04T09:44:52.7531677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7532072Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7532447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7532814Z layer_outputs = layer_module( 2025-12-04T09:44:52.7533160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7533529Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7533910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7534290Z return func(*args, **kwargs) 2025-12-04T09:44:52.7534820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7535450Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7536079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7536560Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7537080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.7537572Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.7537744Z 2025-12-04T09:44:52.7537911Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7538337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7538755Z res = mod(**inputs) 2025-12-04T09:44:52.7539224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7558804Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7559370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7559967Z layer_outputs = layer_module( 2025-12-04T09:44:52.7560454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7560860Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7561299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7561726Z return func(*args, **kwargs) 2025-12-04T09:44:52.7562124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7562584Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7563015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7563431Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7563851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.7564234Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.7564374Z 2025-12-04T09:44:52.7564493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7564857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7565182Z res = mod(**inputs) 2025-12-04T09:44:52.7565544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7565920Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7566297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7566673Z layer_outputs = layer_module( 2025-12-04T09:44:52.7567026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7567392Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7567775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7568141Z return func(*args, **kwargs) 2025-12-04T09:44:52.7568505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7568887Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7569272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7569636Z return func(*args, **kwargs) 2025-12-04T09:44:52.7569996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7570383Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7570759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7571138Z return func(*args, **kwargs) 2025-12-04T09:44:52.7571502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7571868Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7572000Z 2025-12-04T09:44:52.7572103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7572454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7572766Z res = mod(**inputs) 2025-12-04T09:44:52.7573135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7573500Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7573879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7574240Z layer_outputs = layer_module( 2025-12-04T09:44:52.7574593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7574947Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7575317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7575681Z return func(*args, **kwargs) 2025-12-04T09:44:52.7576026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7576398Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7576786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7577140Z return func(*args, **kwargs) 2025-12-04T09:44:52.7577490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7577862Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7578232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7578584Z return func(*args, **kwargs) 2025-12-04T09:44:52.7578931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7579299Z key_states = self.k(current_states) 2025-12-04T09:44:52.7579430Z 2025-12-04T09:44:52.7579533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7579889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7580203Z res = mod(**inputs) 2025-12-04T09:44:52.7580544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7580904Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7581267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7581635Z layer_outputs = layer_module( 2025-12-04T09:44:52.7581965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7582318Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7582687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7583055Z return func(*args, **kwargs) 2025-12-04T09:44:52.7583408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7583784Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7584172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7584551Z return func(*args, **kwargs) 2025-12-04T09:44:52.7584892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7585267Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7585634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7585987Z return func(*args, **kwargs) 2025-12-04T09:44:52.7586335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7586726Z value_states = self.v(current_states) 2025-12-04T09:44:52.7586858Z 2025-12-04T09:44:52.7586943Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7587163Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7587396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7587770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7588075Z res = mod(**inputs) 2025-12-04T09:44:52.7588416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7588784Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7589155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7589517Z layer_outputs = layer_module( 2025-12-04T09:44:52.7589859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7590242Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7590609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7590978Z return func(*args, **kwargs) 2025-12-04T09:44:52.7591340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7591720Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7592093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7592464Z return func(*args, **kwargs) 2025-12-04T09:44:52.7592833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7593222Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7593611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7593988Z return func(*args, **kwargs) 2025-12-04T09:44:52.7594354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7594740Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7594877Z 2025-12-04T09:44:52.7594980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7595337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7595658Z res = mod(**inputs) 2025-12-04T09:44:52.7595999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7596375Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7596747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7597114Z layer_outputs = layer_module( 2025-12-04T09:44:52.7597469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7597840Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7598227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7598596Z return func(*args, **kwargs) 2025-12-04T09:44:52.7598965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7599352Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7599737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7600121Z return func(*args, **kwargs) 2025-12-04T09:44:52.7600651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T09:44:52.7601146Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:44:52.7601376Z 2025-12-04T09:44:52.7601503Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7601922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7602284Z res = mod(**inputs) 2025-12-04T09:44:52.7602672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7603073Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7603445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7603819Z layer_outputs = layer_module( 2025-12-04T09:44:52.7604160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7604544Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7604920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7605289Z return func(*args, **kwargs) 2025-12-04T09:44:52.7605649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7606042Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7606426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7606834Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7607244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.7607642Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.7607791Z 2025-12-04T09:44:52.7607901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7608251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7608570Z res = mod(**inputs) 2025-12-04T09:44:52.7608917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7609288Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7609645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7610015Z layer_outputs = layer_module( 2025-12-04T09:44:52.7610360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7610715Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7611094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7611459Z return func(*args, **kwargs) 2025-12-04T09:44:52.7611818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7612204Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7612601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7613029Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7613438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.7613812Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.7613952Z 2025-12-04T09:44:52.7614064Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7614450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7614768Z res = mod(**inputs) 2025-12-04T09:44:52.7615125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7615485Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7615861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7616229Z layer_outputs = layer_module( 2025-12-04T09:44:52.7616561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7616923Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7617289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7617710Z return func(*args, **kwargs) 2025-12-04T09:44:52.7618218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7618785Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7619223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7619648Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7620059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.7620453Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.7620596Z 2025-12-04T09:44:52.7620719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7621068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7621386Z res = mod(**inputs) 2025-12-04T09:44:52.7621737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7622108Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7622472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7622849Z layer_outputs = layer_module( 2025-12-04T09:44:52.7623206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7623557Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7623935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7624303Z return func(*args, **kwargs) 2025-12-04T09:44:52.7624661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7625042Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7625431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7625842Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7626249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.7626616Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.7626754Z 2025-12-04T09:44:52.7626856Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7627208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7627515Z res = mod(**inputs) 2025-12-04T09:44:52.7627863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7628236Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7628625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7628999Z layer_outputs = layer_module( 2025-12-04T09:44:52.7629332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7629701Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7630059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7630419Z return func(*args, **kwargs) 2025-12-04T09:44:52.7630767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7631147Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7631529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7631914Z return func(*args, **kwargs) 2025-12-04T09:44:52.7632268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7632645Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7633010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7633368Z return func(*args, **kwargs) 2025-12-04T09:44:52.7633717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7634084Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7634227Z 2025-12-04T09:44:52.7634332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7634686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7635004Z res = mod(**inputs) 2025-12-04T09:44:52.7635346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7635718Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7636084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7636455Z layer_outputs = layer_module( 2025-12-04T09:44:52.7636808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7637182Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7637568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7637939Z return func(*args, **kwargs) 2025-12-04T09:44:52.7638319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7638735Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7639138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7639538Z return func(*args, **kwargs) 2025-12-04T09:44:52.7639931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7640416Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7640846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7641264Z return func(*args, **kwargs) 2025-12-04T09:44:52.7641667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7642103Z key_states = self.k(current_states) 2025-12-04T09:44:52.7642248Z 2025-12-04T09:44:52.7642386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7642789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7643160Z res = mod(**inputs) 2025-12-04T09:44:52.7643530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7643961Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7644364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7644774Z layer_outputs = layer_module( 2025-12-04T09:44:52.7645142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7645534Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7645940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7646352Z return func(*args, **kwargs) 2025-12-04T09:44:52.7646732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7647252Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7647655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7648050Z return func(*args, **kwargs) 2025-12-04T09:44:52.7648445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7648873Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7649253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7649612Z return func(*args, **kwargs) 2025-12-04T09:44:52.7649976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7650387Z value_states = self.v(current_states) 2025-12-04T09:44:52.7650534Z 2025-12-04T09:44:52.7650624Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7650860Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7651121Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7651511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7651849Z res = mod(**inputs) 2025-12-04T09:44:52.7652226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7652652Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7653044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7653469Z layer_outputs = layer_module( 2025-12-04T09:44:52.7653846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7654241Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7654643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7655062Z return func(*args, **kwargs) 2025-12-04T09:44:52.7655452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7655856Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7656267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7656684Z return func(*args, **kwargs) 2025-12-04T09:44:52.7657051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7657467Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7657849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7658242Z return func(*args, **kwargs) 2025-12-04T09:44:52.7658616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7658995Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7659133Z 2025-12-04T09:44:52.7659238Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7659597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7659912Z res = mod(**inputs) 2025-12-04T09:44:52.7660261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7660638Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7661056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7661418Z layer_outputs = layer_module( 2025-12-04T09:44:52.7661764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7662129Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7662499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7662869Z return func(*args, **kwargs) 2025-12-04T09:44:52.7663228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7663616Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7663996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7664412Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7664825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.7665217Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.7665375Z 2025-12-04T09:44:52.7665481Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7665839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7666159Z res = mod(**inputs) 2025-12-04T09:44:52.7666502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7666874Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7667246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7667616Z layer_outputs = layer_module( 2025-12-04T09:44:52.7667965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7668338Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7668724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7669121Z return func(*args, **kwargs) 2025-12-04T09:44:52.7669511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7669923Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7670322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7670738Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7671179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.7671584Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.7671716Z 2025-12-04T09:44:52.7671843Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7672195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7672528Z res = mod(**inputs) 2025-12-04T09:44:52.7672882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7673258Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7673637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7674010Z layer_outputs = layer_module( 2025-12-04T09:44:52.7674353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7674732Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7675119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7675501Z return func(*args, **kwargs) 2025-12-04T09:44:52.7675871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7676259Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7676653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7677077Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7677491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.7677893Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.7678047Z 2025-12-04T09:44:52.7678156Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7678518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7678848Z res = mod(**inputs) 2025-12-04T09:44:52.7679194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7679582Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7679959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7680400Z layer_outputs = layer_module( 2025-12-04T09:44:52.7680770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7681176Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7681601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7682009Z return func(*args, **kwargs) 2025-12-04T09:44:52.7682378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7682773Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7683171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7683587Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7684006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.7684391Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.7684527Z 2025-12-04T09:44:52.7684633Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7684997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7685347Z res = mod(**inputs) 2025-12-04T09:44:52.7685703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7686098Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7686487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7686868Z layer_outputs = layer_module( 2025-12-04T09:44:52.7687222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7687586Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7687975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7688357Z return func(*args, **kwargs) 2025-12-04T09:44:52.7688719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7689131Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7689526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7689910Z return func(*args, **kwargs) 2025-12-04T09:44:52.7690277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7690672Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7691063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7691438Z return func(*args, **kwargs) 2025-12-04T09:44:52.7691816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7692194Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7692330Z 2025-12-04T09:44:52.7692445Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7692797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7693121Z res = mod(**inputs) 2025-12-04T09:44:52.7693472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7693841Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7694214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7694588Z layer_outputs = layer_module( 2025-12-04T09:44:52.7694933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7695290Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7695700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7696108Z return func(*args, **kwargs) 2025-12-04T09:44:52.7696475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7696862Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7697265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7697631Z return func(*args, **kwargs) 2025-12-04T09:44:52.7697984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7698367Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7698750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7699121Z return func(*args, **kwargs) 2025-12-04T09:44:52.7699501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7699890Z key_states = self.k(current_states) 2025-12-04T09:44:52.7700048Z 2025-12-04T09:44:52.7700164Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7700544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7700886Z res = mod(**inputs) 2025-12-04T09:44:52.7701231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7701605Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7701963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7702331Z layer_outputs = layer_module( 2025-12-04T09:44:52.7702677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7703056Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7703428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7703808Z return func(*args, **kwargs) 2025-12-04T09:44:52.7704179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7704563Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7704957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7705333Z return func(*args, **kwargs) 2025-12-04T09:44:52.7705703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7706094Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7706481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7706853Z return func(*args, **kwargs) 2025-12-04T09:44:52.7707211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7707600Z value_states = self.v(current_states) 2025-12-04T09:44:52.7707746Z 2025-12-04T09:44:52.7707829Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7708053Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7708287Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7708659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7708980Z res = mod(**inputs) 2025-12-04T09:44:52.7709329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7709711Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7710091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7710465Z layer_outputs = layer_module( 2025-12-04T09:44:52.7710804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7711169Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7711546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7711912Z return func(*args, **kwargs) 2025-12-04T09:44:52.7712284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7712675Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7713087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7713462Z return func(*args, **kwargs) 2025-12-04T09:44:52.7713828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7714242Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7714657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7715031Z return func(*args, **kwargs) 2025-12-04T09:44:52.7715397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7715780Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7715912Z 2025-12-04T09:44:52.7716019Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7716391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7716735Z res = mod(**inputs) 2025-12-04T09:44:52.7717138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7717537Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7717936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7718351Z layer_outputs = layer_module( 2025-12-04T09:44:52.7718719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7719112Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7719518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7719926Z return func(*args, **kwargs) 2025-12-04T09:44:52.7720303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7720787Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7721205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7721622Z return func(*args, **kwargs) 2025-12-04T09:44:52.7722026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T09:44:52.7722496Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:44:52.7722696Z 2025-12-04T09:44:52.7722815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7723195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7723542Z res = mod(**inputs) 2025-12-04T09:44:52.7723924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7724341Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7724734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7725140Z layer_outputs = layer_module( 2025-12-04T09:44:52.7725520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7725885Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7726273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7726649Z return func(*args, **kwargs) 2025-12-04T09:44:52.7727017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7727408Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7727830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7728260Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7728680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.7729108Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.7729288Z 2025-12-04T09:44:52.7729395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7729759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7730075Z res = mod(**inputs) 2025-12-04T09:44:52.7730439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7730841Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7731235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7731628Z layer_outputs = layer_module( 2025-12-04T09:44:52.7731983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7732370Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7732781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7733160Z return func(*args, **kwargs) 2025-12-04T09:44:52.7733529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7733928Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7734321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7734748Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7735169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.7735560Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.7735699Z 2025-12-04T09:44:52.7735805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7736174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7736503Z res = mod(**inputs) 2025-12-04T09:44:52.7736848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7737233Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7737611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7737989Z layer_outputs = layer_module( 2025-12-04T09:44:52.7738335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7738703Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7739086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7739457Z return func(*args, **kwargs) 2025-12-04T09:44:52.7739825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7740220Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7740614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7741034Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7741453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.7741851Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.7742018Z 2025-12-04T09:44:52.7742537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7742898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7743266Z res = mod(**inputs) 2025-12-04T09:44:52.7743671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7744048Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7744424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7744806Z layer_outputs = layer_module( 2025-12-04T09:44:52.7745159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7745523Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7745910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7746310Z return func(*args, **kwargs) 2025-12-04T09:44:52.7746707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7747325Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7747772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7748237Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7748681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.7749108Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.7749262Z 2025-12-04T09:44:52.7749377Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7749767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7750120Z res = mod(**inputs) 2025-12-04T09:44:52.7750506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7750914Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7751318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7751730Z layer_outputs = layer_module( 2025-12-04T09:44:52.7752102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7752499Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7752908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7753325Z return func(*args, **kwargs) 2025-12-04T09:44:52.7753711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7754128Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7754538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7754942Z return func(*args, **kwargs) 2025-12-04T09:44:52.7755325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7755734Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7756156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7756565Z return func(*args, **kwargs) 2025-12-04T09:44:52.7756951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7757393Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7757545Z 2025-12-04T09:44:52.7757657Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7758076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7758423Z res = mod(**inputs) 2025-12-04T09:44:52.7758846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7759258Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7759652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7760050Z layer_outputs = layer_module( 2025-12-04T09:44:52.7760488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7760893Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7761302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7761749Z return func(*args, **kwargs) 2025-12-04T09:44:52.7762150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7762559Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7762967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7763348Z return func(*args, **kwargs) 2025-12-04T09:44:52.7763719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7764101Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7764482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7764869Z return func(*args, **kwargs) 2025-12-04T09:44:52.7765249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7765632Z key_states = self.k(current_states) 2025-12-04T09:44:52.7765786Z 2025-12-04T09:44:52.7765893Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7766268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7766595Z res = mod(**inputs) 2025-12-04T09:44:52.7766944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7767330Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7767708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7768088Z layer_outputs = layer_module( 2025-12-04T09:44:52.7768442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7768825Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7769219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7769588Z return func(*args, **kwargs) 2025-12-04T09:44:52.7769966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7770416Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7770815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7771191Z return func(*args, **kwargs) 2025-12-04T09:44:52.7771552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7771965Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7772358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7772765Z return func(*args, **kwargs) 2025-12-04T09:44:52.7773153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7773543Z value_states = self.v(current_states) 2025-12-04T09:44:52.7773688Z 2025-12-04T09:44:52.7773773Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7773977Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7774205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7774561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7774880Z res = mod(**inputs) 2025-12-04T09:44:52.7775212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7775598Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7775978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7776364Z layer_outputs = layer_module( 2025-12-04T09:44:52.7776706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7777067Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7777450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7777800Z return func(*args, **kwargs) 2025-12-04T09:44:52.7778147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7778514Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7778884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7779247Z return func(*args, **kwargs) 2025-12-04T09:44:52.7779603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7779982Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7780352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7780709Z return func(*args, **kwargs) 2025-12-04T09:44:52.7781074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7781429Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7781558Z 2025-12-04T09:44:52.7781662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7782010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7782323Z res = mod(**inputs) 2025-12-04T09:44:52.7782659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7783036Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7783403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7783769Z layer_outputs = layer_module( 2025-12-04T09:44:52.7784114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7784464Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7784840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7785212Z return func(*args, **kwargs) 2025-12-04T09:44:52.7785602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7786003Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7786407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7786835Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7787246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.7787647Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.7787797Z 2025-12-04T09:44:52.7787899Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7788254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7788577Z res = mod(**inputs) 2025-12-04T09:44:52.7788934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7789332Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7789695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7790063Z layer_outputs = layer_module( 2025-12-04T09:44:52.7790399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7790756Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7791133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7791499Z return func(*args, **kwargs) 2025-12-04T09:44:52.7791851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7792243Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7792646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7793069Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7793502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.7793917Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.7794062Z 2025-12-04T09:44:52.7794174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7794559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7794889Z res = mod(**inputs) 2025-12-04T09:44:52.7795244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7795628Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7796021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7796431Z layer_outputs = layer_module( 2025-12-04T09:44:52.7796803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7797190Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7797608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7798013Z return func(*args, **kwargs) 2025-12-04T09:44:52.7798399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7798810Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7799234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7799707Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7800154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.7800654Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.7800814Z 2025-12-04T09:44:52.7800926Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7801335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7801672Z res = mod(**inputs) 2025-12-04T09:44:52.7802027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7802415Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7802781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7803149Z layer_outputs = layer_module( 2025-12-04T09:44:52.7804378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7804735Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7805103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7805479Z return func(*args, **kwargs) 2025-12-04T09:44:52.7805840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7806230Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7806608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7807017Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7807426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.7807808Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.7807945Z 2025-12-04T09:44:52.7808051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7808425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7808746Z res = mod(**inputs) 2025-12-04T09:44:52.7809085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7809459Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7809830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7810201Z layer_outputs = layer_module( 2025-12-04T09:44:52.7810541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7810905Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7811281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7811643Z return func(*args, **kwargs) 2025-12-04T09:44:52.7812003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7812378Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7812754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7813112Z return func(*args, **kwargs) 2025-12-04T09:44:52.7813466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7813849Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7814221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7814620Z return func(*args, **kwargs) 2025-12-04T09:44:52.7814979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7815380Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7815513Z 2025-12-04T09:44:52.7815639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7816001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7816323Z res = mod(**inputs) 2025-12-04T09:44:52.7816664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7817047Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7817430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7817811Z layer_outputs = layer_module( 2025-12-04T09:44:52.7818183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7818559Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7818949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7819330Z return func(*args, **kwargs) 2025-12-04T09:44:52.7819698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7820095Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7820486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7820860Z return func(*args, **kwargs) 2025-12-04T09:44:52.7821235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7821631Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7822025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7822401Z return func(*args, **kwargs) 2025-12-04T09:44:52.7822773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7823164Z key_states = self.k(current_states) 2025-12-04T09:44:52.7823301Z 2025-12-04T09:44:52.7823410Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7823785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7824116Z res = mod(**inputs) 2025-12-04T09:44:52.7824480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7824863Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7825249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7825635Z layer_outputs = layer_module( 2025-12-04T09:44:52.7825995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7826363Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7826754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7827141Z return func(*args, **kwargs) 2025-12-04T09:44:52.7827507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7827902Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7828298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7828724Z return func(*args, **kwargs) 2025-12-04T09:44:52.7829087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7829497Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7829898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7830270Z return func(*args, **kwargs) 2025-12-04T09:44:52.7830635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7831019Z value_states = self.v(current_states) 2025-12-04T09:44:52.7831162Z 2025-12-04T09:44:52.7831252Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7831462Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7831699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7832060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7832399Z res = mod(**inputs) 2025-12-04T09:44:52.7832748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7833127Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7833532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7833907Z layer_outputs = layer_module( 2025-12-04T09:44:52.7834260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7834627Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7835008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7835385Z return func(*args, **kwargs) 2025-12-04T09:44:52.7835753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7836144Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7836528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7836905Z return func(*args, **kwargs) 2025-12-04T09:44:52.7837270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7837661Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7838065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7838464Z return func(*args, **kwargs) 2025-12-04T09:44:52.7838851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7839253Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7839407Z 2025-12-04T09:44:52.7839519Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7839917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7840276Z res = mod(**inputs) 2025-12-04T09:44:52.7840738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7841156Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7841583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7841988Z layer_outputs = layer_module( 2025-12-04T09:44:52.7842364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7842765Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7843205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7843609Z return func(*args, **kwargs) 2025-12-04T09:44:52.7844020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7844446Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7844834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7845201Z return func(*args, **kwargs) 2025-12-04T09:44:52.7845570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7845959Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7846361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7846791Z return func(*args, **kwargs) 2025-12-04T09:44:52.7847356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7847777Z key_states = self.k(current_states) 2025-12-04T09:44:52.7847921Z 2025-12-04T09:44:52.7848035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7848427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7848775Z res = mod(**inputs) 2025-12-04T09:44:52.7849143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7849569Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7849969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7850385Z layer_outputs = layer_module( 2025-12-04T09:44:52.7850758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7851159Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7851571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7851971Z return func(*args, **kwargs) 2025-12-04T09:44:52.7852361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7852774Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7853181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7853576Z return func(*args, **kwargs) 2025-12-04T09:44:52.7853961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7854378Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7854789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7855183Z return func(*args, **kwargs) 2025-12-04T09:44:52.7855572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7855982Z value_states = self.v(current_states) 2025-12-04T09:44:52.7856128Z 2025-12-04T09:44:52.7856215Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7856447Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7856703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7857089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7857434Z res = mod(**inputs) 2025-12-04T09:44:52.7857844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7858254Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7858649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7859056Z layer_outputs = layer_module( 2025-12-04T09:44:52.7859443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7859819Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7860199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7860576Z return func(*args, **kwargs) 2025-12-04T09:44:52.7860943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7861327Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7861719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7862128Z return func(*args, **kwargs) 2025-12-04T09:44:52.7862504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7862893Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7863289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7863672Z return func(*args, **kwargs) 2025-12-04T09:44:52.7864030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7864420Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7864564Z 2025-12-04T09:44:52.7864673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7865049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7865380Z res = mod(**inputs) 2025-12-04T09:44:52.7865744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7866138Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7866525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7866909Z layer_outputs = layer_module( 2025-12-04T09:44:52.7867273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7867647Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7868026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7868405Z return func(*args, **kwargs) 2025-12-04T09:44:52.7868779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.7869178Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.7869570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7869953Z return func(*args, **kwargs) 2025-12-04T09:44:52.7870326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.7870721Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.7871135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7871548Z return func(*args, **kwargs) 2025-12-04T09:44:52.7871918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7872325Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7872471Z 2025-12-04T09:44:52.7872580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7872953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7873294Z res = mod(**inputs) 2025-12-04T09:44:52.7873660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7874044Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7874423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7874819Z layer_outputs = layer_module( 2025-12-04T09:44:52.7875197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7875579Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7875967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7876375Z return func(*args, **kwargs) 2025-12-04T09:44:52.7876763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7877175Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7877579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7877979Z return func(*args, **kwargs) 2025-12-04T09:44:52.7878365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7878775Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7879177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7879577Z return func(*args, **kwargs) 2025-12-04T09:44:52.7879967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7880441Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7880590Z 2025-12-04T09:44:52.7880710Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7881111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7881476Z res = mod(**inputs) 2025-12-04T09:44:52.7881853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7882263Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7882673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7883082Z layer_outputs = layer_module( 2025-12-04T09:44:52.7883460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7883859Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7884274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7884692Z return func(*args, **kwargs) 2025-12-04T09:44:52.7885084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7885498Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7885915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7886309Z return func(*args, **kwargs) 2025-12-04T09:44:52.7886702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T09:44:52.7887199Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:44:52.7887402Z 2025-12-04T09:44:52.7887520Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7887923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7888268Z res = mod(**inputs) 2025-12-04T09:44:52.7888659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7889060Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7889457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7889858Z layer_outputs = layer_module( 2025-12-04T09:44:52.7890228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7890614Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7891014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7891379Z return func(*args, **kwargs) 2025-12-04T09:44:52.7891741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7892148Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7892546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7892973Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7893385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.7893796Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.7893957Z 2025-12-04T09:44:52.7894068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7894430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7894738Z res = mod(**inputs) 2025-12-04T09:44:52.7895087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7895473Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7895850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7896224Z layer_outputs = layer_module( 2025-12-04T09:44:52.7896569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7896933Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7897298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7897673Z return func(*args, **kwargs) 2025-12-04T09:44:52.7898043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7898440Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7898829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7899247Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7899658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.7900031Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.7900174Z 2025-12-04T09:44:52.7900279Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7900635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7900958Z res = mod(**inputs) 2025-12-04T09:44:52.7901332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7901705Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7902107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7902502Z layer_outputs = layer_module( 2025-12-04T09:44:52.7902865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7903238Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7903625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7903999Z return func(*args, **kwargs) 2025-12-04T09:44:52.7904383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7904796Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7905176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7905594Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7906002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.7906388Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.7906528Z 2025-12-04T09:44:52.7906632Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7906988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7907308Z res = mod(**inputs) 2025-12-04T09:44:52.7907658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:44:52.7908050Z encoder_outputs = self.encoder( 2025-12-04T09:44:52.7908422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7908804Z layer_outputs = layer_module( 2025-12-04T09:44:52.7909152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7909531Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7909917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7910298Z return func(*args, **kwargs) 2025-12-04T09:44:52.7910659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7911063Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7911463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7911883Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7912310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.7912702Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.7912838Z 2025-12-04T09:44:52.7912955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7913320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7913651Z res = mod(**inputs) 2025-12-04T09:44:52.7914010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7914404Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7914780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7915194Z layer_outputs = layer_module( 2025-12-04T09:44:52.7915559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7915947Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7916376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7916759Z return func(*args, **kwargs) 2025-12-04T09:44:52.7917143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.7917554Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.7917963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7918363Z return func(*args, **kwargs) 2025-12-04T09:44:52.7918746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.7919190Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.7919603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7920003Z return func(*args, **kwargs) 2025-12-04T09:44:52.7920459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7920888Z key_states = self.k(current_states) 2025-12-04T09:44:52.7921040Z 2025-12-04T09:44:52.7921167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7921572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7921920Z res = mod(**inputs) 2025-12-04T09:44:52.7922304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7922693Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7923061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7923456Z layer_outputs = layer_module( 2025-12-04T09:44:52.7923838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7924227Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7924624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7925026Z return func(*args, **kwargs) 2025-12-04T09:44:52.7925415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.7925816Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.7926228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7926635Z return func(*args, **kwargs) 2025-12-04T09:44:52.7927001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.7927387Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.7927786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7928186Z return func(*args, **kwargs) 2025-12-04T09:44:52.7928566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7928974Z value_states = self.v(current_states) 2025-12-04T09:44:52.7929128Z 2025-12-04T09:44:52.7929216Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7929453Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7929707Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7930134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7930486Z res = mod(**inputs) 2025-12-04T09:44:52.7930882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7931344Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7931747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7932157Z layer_outputs = layer_module( 2025-12-04T09:44:52.7932525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7932913Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7933319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7933723Z return func(*args, **kwargs) 2025-12-04T09:44:52.7934135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.7934552Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.7934961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7935351Z return func(*args, **kwargs) 2025-12-04T09:44:52.7935739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.7936155Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.7936564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7936955Z return func(*args, **kwargs) 2025-12-04T09:44:52.7937346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7937759Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7937898Z 2025-12-04T09:44:52.7938011Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7938405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7938749Z res = mod(**inputs) 2025-12-04T09:44:52.7939130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7939530Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7939929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7940329Z layer_outputs = layer_module( 2025-12-04T09:44:52.7940697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7941095Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7941498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7941878Z return func(*args, **kwargs) 2025-12-04T09:44:52.7942238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7942639Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7943038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7943462Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7943887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.7944321Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.7944473Z 2025-12-04T09:44:52.7944611Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7944975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7945324Z res = mod(**inputs) 2025-12-04T09:44:52.7945684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7946087Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7946462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7946850Z layer_outputs = layer_module( 2025-12-04T09:44:52.7947455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7947852Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7948263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7948708Z return func(*args, **kwargs) 2025-12-04T09:44:52.7949086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7949480Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7949878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7950304Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7950725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.7951110Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.7951253Z 2025-12-04T09:44:52.7951359Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7951727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7952045Z res = mod(**inputs) 2025-12-04T09:44:52.7952400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7952784Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7953164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7953545Z layer_outputs = layer_module( 2025-12-04T09:44:52.7953902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7954272Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7954652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7955034Z return func(*args, **kwargs) 2025-12-04T09:44:52.7955400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7955813Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7956224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7956676Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7957123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.7957219Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.7957223Z 2025-12-04T09:44:52.7957340Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7957551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7957621Z res = mod(**inputs) 2025-12-04T09:44:52.7957890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7958002Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7958270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7958371Z layer_outputs = layer_module( 2025-12-04T09:44:52.7958637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7958735Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7958997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7959071Z return func(*args, **kwargs) 2025-12-04T09:44:52.7959336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7959434Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7959698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7959844Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7960106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.7960201Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.7960207Z 2025-12-04T09:44:52.7960362Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7960597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7960669Z res = mod(**inputs) 2025-12-04T09:44:52.7960929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7961014Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7961284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7961367Z layer_outputs = layer_module( 2025-12-04T09:44:52.7961623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7961720Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7961986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7962060Z return func(*args, **kwargs) 2025-12-04T09:44:52.7962314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7962409Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7962667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7962741Z return func(*args, **kwargs) 2025-12-04T09:44:52.7963008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7963101Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7963367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7963439Z return func(*args, **kwargs) 2025-12-04T09:44:52.7963694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7963786Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7963790Z 2025-12-04T09:44:52.7963903Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7964122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7964191Z res = mod(**inputs) 2025-12-04T09:44:52.7964476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7964566Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7964826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7964920Z layer_outputs = layer_module( 2025-12-04T09:44:52.7965184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7965272Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7965540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7965614Z return func(*args, **kwargs) 2025-12-04T09:44:52.7965875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7965969Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7966227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7966325Z return func(*args, **kwargs) 2025-12-04T09:44:52.7966600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7966691Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7966958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7967030Z return func(*args, **kwargs) 2025-12-04T09:44:52.7967287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7967379Z key_states = self.k(current_states) 2025-12-04T09:44:52.7967383Z 2025-12-04T09:44:52.7967493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7967717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7967788Z res = mod(**inputs) 2025-12-04T09:44:52.7968047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7968132Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7968376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7968448Z layer_outputs = layer_module( 2025-12-04T09:44:52.7968680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7968759Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7969008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7969078Z return func(*args, **kwargs) 2025-12-04T09:44:52.7969317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7969406Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7969649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7969718Z return func(*args, **kwargs) 2025-12-04T09:44:52.7969966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7970048Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7970295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7970364Z return func(*args, **kwargs) 2025-12-04T09:44:52.7970602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7970688Z value_states = self.v(current_states) 2025-12-04T09:44:52.7970722Z 2025-12-04T09:44:52.7970809Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7970889Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7971032Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7971232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7971320Z res = mod(**inputs) 2025-12-04T09:44:52.7971573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7971647Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7971898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7971970Z layer_outputs = layer_module( 2025-12-04T09:44:52.7972201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7972302Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7972547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7972627Z return func(*args, **kwargs) 2025-12-04T09:44:52.7972873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.7972957Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.7973209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7973278Z return func(*args, **kwargs) 2025-12-04T09:44:52.7973526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.7973610Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.7973860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7973937Z return func(*args, **kwargs) 2025-12-04T09:44:52.7974179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7974258Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7974270Z 2025-12-04T09:44:52.7974376Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7974578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7974651Z res = mod(**inputs) 2025-12-04T09:44:52.7974893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7974966Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7975220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7975295Z layer_outputs = layer_module( 2025-12-04T09:44:52.7975526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7975606Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7975849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7975924Z return func(*args, **kwargs) 2025-12-04T09:44:52.7976165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.7976246Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.7976497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7976564Z return func(*args, **kwargs) 2025-12-04T09:44:52.7976837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.7976926Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.7977170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7977270Z return func(*args, **kwargs) 2025-12-04T09:44:52.7977530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.7977611Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.7977622Z 2025-12-04T09:44:52.7977726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7977926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7977996Z res = mod(**inputs) 2025-12-04T09:44:52.7978241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7978338Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7978590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7978663Z layer_outputs = layer_module( 2025-12-04T09:44:52.7978907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7978993Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7979253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7979333Z return func(*args, **kwargs) 2025-12-04T09:44:52.7979593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.7979679Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.7979948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7980021Z return func(*args, **kwargs) 2025-12-04T09:44:52.7980283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.7980370Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.7980618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7980695Z return func(*args, **kwargs) 2025-12-04T09:44:52.7980942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.7981022Z key_states = self.k(current_states) 2025-12-04T09:44:52.7981033Z 2025-12-04T09:44:52.7981143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7981350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7981428Z res = mod(**inputs) 2025-12-04T09:44:52.7981675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7981753Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7982006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7982079Z layer_outputs = layer_module( 2025-12-04T09:44:52.7982314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7982394Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7982639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7982714Z return func(*args, **kwargs) 2025-12-04T09:44:52.7982979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.7983064Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.7983325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7983418Z return func(*args, **kwargs) 2025-12-04T09:44:52.7983699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.7983789Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.7984045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7984123Z return func(*args, **kwargs) 2025-12-04T09:44:52.7984365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.7984442Z value_states = self.v(current_states) 2025-12-04T09:44:52.7984478Z 2025-12-04T09:44:52.7984560Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7984639Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.7984752Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7984953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7985020Z res = mod(**inputs) 2025-12-04T09:44:52.7985274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7985348Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7985595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7985674Z layer_outputs = layer_module( 2025-12-04T09:44:52.7985896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7985986Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7986229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7986299Z return func(*args, **kwargs) 2025-12-04T09:44:52.7986551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.7986631Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.7986881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7986950Z return func(*args, **kwargs) 2025-12-04T09:44:52.7987191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.7987282Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.7987533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7987603Z return func(*args, **kwargs) 2025-12-04T09:44:52.7987855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.7987935Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.7987939Z 2025-12-04T09:44:52.7988051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7988250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7988317Z res = mod(**inputs) 2025-12-04T09:44:52.7988569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7988643Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7988885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7988984Z layer_outputs = layer_module( 2025-12-04T09:44:52.7989213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7989318Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7989579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7989648Z return func(*args, **kwargs) 2025-12-04T09:44:52.7989900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7989993Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7990245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7990362Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7990595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.7990723Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.7990728Z 2025-12-04T09:44:52.7990830Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7991030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7991100Z res = mod(**inputs) 2025-12-04T09:44:52.7991345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7991426Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7991678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7991748Z layer_outputs = layer_module( 2025-12-04T09:44:52.7991980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7992061Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7992312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7992383Z return func(*args, **kwargs) 2025-12-04T09:44:52.7992626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7992724Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7992964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7993085Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7993331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.7993410Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.7993415Z 2025-12-04T09:44:52.7993525Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7993725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7993791Z res = mod(**inputs) 2025-12-04T09:44:52.7994045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7994117Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7994368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7994439Z layer_outputs = layer_module( 2025-12-04T09:44:52.7994664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7994749Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7995022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7995094Z return func(*args, **kwargs) 2025-12-04T09:44:52.7995344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7995451Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7995709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7995824Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7996059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.7996158Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.7996161Z 2025-12-04T09:44:52.7996265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7996466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7996559Z res = mod(**inputs) 2025-12-04T09:44:52.7996806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7996890Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7997137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7997209Z layer_outputs = layer_module( 2025-12-04T09:44:52.7997441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.7997522Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.7997778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.7997848Z return func(*args, **kwargs) 2025-12-04T09:44:52.7998092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.7998191Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.7998437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.7998556Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.7998811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.7998892Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.7998897Z 2025-12-04T09:44:52.7999007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.7999209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.7999274Z res = mod(**inputs) 2025-12-04T09:44:52.7999531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.7999605Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.7999873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.7999950Z layer_outputs = layer_module( 2025-12-04T09:44:52.8000189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8000280Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8000711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8000790Z return func(*args, **kwargs) 2025-12-04T09:44:52.8001057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8001146Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8001443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8001520Z return func(*args, **kwargs) 2025-12-04T09:44:52.8001795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8001914Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8002174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8002248Z return func(*args, **kwargs) 2025-12-04T09:44:52.8002513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8002597Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8002601Z 2025-12-04T09:44:52.8002719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8002940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8003029Z res = mod(**inputs) 2025-12-04T09:44:52.8003275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8003349Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8003594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8003665Z layer_outputs = layer_module( 2025-12-04T09:44:52.8003881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8003966Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8004210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8004280Z return func(*args, **kwargs) 2025-12-04T09:44:52.8004531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8004615Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8004866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8004937Z return func(*args, **kwargs) 2025-12-04T09:44:52.8005177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8005269Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8005511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8005578Z return func(*args, **kwargs) 2025-12-04T09:44:52.8005826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8005907Z key_states = self.k(current_states) 2025-12-04T09:44:52.8005911Z 2025-12-04T09:44:52.8006022Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8006224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8006289Z res = mod(**inputs) 2025-12-04T09:44:52.8006541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8006615Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8006864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8006935Z layer_outputs = layer_module( 2025-12-04T09:44:52.8007157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8007242Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8007506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8007578Z return func(*args, **kwargs) 2025-12-04T09:44:52.8007852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8007963Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8008211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8008278Z return func(*args, **kwargs) 2025-12-04T09:44:52.8008518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8008609Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8008854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8008945Z return func(*args, **kwargs) 2025-12-04T09:44:52.8009197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8009276Z value_states = self.v(current_states) 2025-12-04T09:44:52.8009280Z 2025-12-04T09:44:52.8009367Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8009449Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8009553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8009764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8009830Z res = mod(**inputs) 2025-12-04T09:44:52.8010078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8010161Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8010410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8010493Z layer_outputs = layer_module( 2025-12-04T09:44:52.8010719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8010801Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8011059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8011128Z return func(*args, **kwargs) 2025-12-04T09:44:52.8011384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8011466Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8011713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8011792Z return func(*args, **kwargs) 2025-12-04T09:44:52.8012040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8012126Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8012379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8012446Z return func(*args, **kwargs) 2025-12-04T09:44:52.8012699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8012778Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8012781Z 2025-12-04T09:44:52.8012885Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8013098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8013163Z res = mod(**inputs) 2025-12-04T09:44:52.8013436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8013520Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8013763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8013873Z layer_outputs = layer_module( 2025-12-04T09:44:52.8014121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8014203Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8014456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8014524Z return func(*args, **kwargs) 2025-12-04T09:44:52.8014781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8014861Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8015104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8015207Z return func(*args, **kwargs) 2025-12-04T09:44:52.8015450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T09:44:52.8015589Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:44:52.8015600Z 2025-12-04T09:44:52.8015705Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8015904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8015978Z res = mod(**inputs) 2025-12-04T09:44:52.8016222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8016295Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8016547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8016620Z layer_outputs = layer_module( 2025-12-04T09:44:52.8016850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8016931Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8017178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8017256Z return func(*args, **kwargs) 2025-12-04T09:44:52.8017501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8017582Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8017833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8017902Z return func(*args, **kwargs) 2025-12-04T09:44:52.8018152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8018239Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8018484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8018563Z return func(*args, **kwargs) 2025-12-04T09:44:52.8018803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8018883Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8018895Z 2025-12-04T09:44:52.8018999Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8019199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8019273Z res = mod(**inputs) 2025-12-04T09:44:52.8019537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8019617Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8019867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8019960Z layer_outputs = layer_module( 2025-12-04T09:44:52.8020205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8020285Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8020532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8020608Z return func(*args, **kwargs) 2025-12-04T09:44:52.8020852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8020934Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8021207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8021277Z return func(*args, **kwargs) 2025-12-04T09:44:52.8021527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8021612Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8021851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8021926Z return func(*args, **kwargs) 2025-12-04T09:44:52.8022167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8022245Z key_states = self.k(current_states) 2025-12-04T09:44:52.8022256Z 2025-12-04T09:44:52.8022369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8022567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8022639Z res = mod(**inputs) 2025-12-04T09:44:52.8022878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8022952Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8023201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8023273Z layer_outputs = layer_module( 2025-12-04T09:44:52.8023505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8023583Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8023826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8023901Z return func(*args, **kwargs) 2025-12-04T09:44:52.8024145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8024231Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8024515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8024589Z return func(*args, **kwargs) 2025-12-04T09:44:52.8024849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8024938Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8025203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8025285Z return func(*args, **kwargs) 2025-12-04T09:44:52.8025549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8025653Z value_states = self.v(current_states) 2025-12-04T09:44:52.8025666Z 2025-12-04T09:44:52.8025755Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8025842Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8025984Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8026220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8026285Z res = mod(**inputs) 2025-12-04T09:44:52.8026537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8026609Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8026852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8026931Z layer_outputs = layer_module( 2025-12-04T09:44:52.8027160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8027269Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8027513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8027585Z return func(*args, **kwargs) 2025-12-04T09:44:52.8027833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8027914Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8028161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8028232Z return func(*args, **kwargs) 2025-12-04T09:44:52.8028473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8028566Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8028812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8028882Z return func(*args, **kwargs) 2025-12-04T09:44:52.8029132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8029210Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8029215Z 2025-12-04T09:44:52.8029324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8029524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8029590Z res = mod(**inputs) 2025-12-04T09:44:52.8029840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8029913Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8030155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8030235Z layer_outputs = layer_module( 2025-12-04T09:44:52.8030466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8030550Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8030787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8030856Z return func(*args, **kwargs) 2025-12-04T09:44:52.8031098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8031190Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8031435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8031551Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8031809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.8031920Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.8031943Z 2025-12-04T09:44:52.8032046Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8032263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8032336Z res = mod(**inputs) 2025-12-04T09:44:52.8032576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8032654Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8032890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8032959Z layer_outputs = layer_module( 2025-12-04T09:44:52.8033190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8033288Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8033537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8033608Z return func(*args, **kwargs) 2025-12-04T09:44:52.8033847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8033953Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8034188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8034298Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8034536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.8034613Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.8034618Z 2025-12-04T09:44:52.8034722Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8034913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8034978Z res = mod(**inputs) 2025-12-04T09:44:52.8035227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8035299Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8035549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8035620Z layer_outputs = layer_module( 2025-12-04T09:44:52.8035844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8035926Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8036170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8036242Z return func(*args, **kwargs) 2025-12-04T09:44:52.8036489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8036581Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8036827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8036942Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8037181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.8037274Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.8037278Z 2025-12-04T09:44:52.8037382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8037601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8037677Z res = mod(**inputs) 2025-12-04T09:44:52.8037921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8038019Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8038291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8038365Z layer_outputs = layer_module( 2025-12-04T09:44:52.8038595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8038676Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8038930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8038998Z return func(*args, **kwargs) 2025-12-04T09:44:52.8039262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8039359Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8039603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8039723Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8039975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.8040061Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.8040066Z 2025-12-04T09:44:52.8040184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8040477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8040553Z res = mod(**inputs) 2025-12-04T09:44:52.8040827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8040909Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8041242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8041320Z layer_outputs = layer_module( 2025-12-04T09:44:52.8041569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8041666Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8041943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8042021Z return func(*args, **kwargs) 2025-12-04T09:44:52.8042308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8042400Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8042690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8042760Z return func(*args, **kwargs) 2025-12-04T09:44:52.8043005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8043101Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8043347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8043416Z return func(*args, **kwargs) 2025-12-04T09:44:52.8043671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8043751Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8043755Z 2025-12-04T09:44:52.8043872Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8044096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8044164Z res = mod(**inputs) 2025-12-04T09:44:52.8044434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8044507Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8044777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8044851Z layer_outputs = layer_module( 2025-12-04T09:44:52.8045074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8045159Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8045405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8045475Z return func(*args, **kwargs) 2025-12-04T09:44:52.8045754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8045837Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8046086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8046156Z return func(*args, **kwargs) 2025-12-04T09:44:52.8046398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8046491Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8046737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8046806Z return func(*args, **kwargs) 2025-12-04T09:44:52.8047174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8047262Z key_states = self.k(current_states) 2025-12-04T09:44:52.8047266Z 2025-12-04T09:44:52.8047381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8047585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8047650Z res = mod(**inputs) 2025-12-04T09:44:52.8047909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8047987Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8048247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8048319Z layer_outputs = layer_module( 2025-12-04T09:44:52.8048547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8048636Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8048891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8048965Z return func(*args, **kwargs) 2025-12-04T09:44:52.8049232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8049322Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8049598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8049668Z return func(*args, **kwargs) 2025-12-04T09:44:52.8049912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8050002Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8050297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8050372Z return func(*args, **kwargs) 2025-12-04T09:44:52.8050620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8050723Z value_states = self.v(current_states) 2025-12-04T09:44:52.8050727Z 2025-12-04T09:44:52.8050835Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8050918Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8051024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8051239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8051304Z res = mod(**inputs) 2025-12-04T09:44:52.8051551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8051632Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8051878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8051986Z layer_outputs = layer_module( 2025-12-04T09:44:52.8052211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8052292Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8052543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8052612Z return func(*args, **kwargs) 2025-12-04T09:44:52.8052864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8052946Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8053190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8053266Z return func(*args, **kwargs) 2025-12-04T09:44:52.8053508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8053592Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8053839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8053909Z return func(*args, **kwargs) 2025-12-04T09:44:52.8054157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8054235Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8054239Z 2025-12-04T09:44:52.8054343Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8054551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8054615Z res = mod(**inputs) 2025-12-04T09:44:52.8054862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8054947Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8055190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8055270Z layer_outputs = layer_module( 2025-12-04T09:44:52.8055496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8055576Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8055828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8055897Z return func(*args, **kwargs) 2025-12-04T09:44:52.8056143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8056226Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8056489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8056568Z return func(*args, **kwargs) 2025-12-04T09:44:52.8056832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8056942Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8057195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8057264Z return func(*args, **kwargs) 2025-12-04T09:44:52.8057511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8057590Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8057594Z 2025-12-04T09:44:52.8057698Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8057906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8057993Z res = mod(**inputs) 2025-12-04T09:44:52.8058236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8058321Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8058577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8058653Z layer_outputs = layer_module( 2025-12-04T09:44:52.8058870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8058946Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8059192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8059259Z return func(*args, **kwargs) 2025-12-04T09:44:52.8059506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8059586Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8059832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8059905Z return func(*args, **kwargs) 2025-12-04T09:44:52.8060134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8060215Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8060453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8060518Z return func(*args, **kwargs) 2025-12-04T09:44:52.8060752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8060828Z key_states = self.k(current_states) 2025-12-04T09:44:52.8060831Z 2025-12-04T09:44:52.8060932Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8061131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8061192Z res = mod(**inputs) 2025-12-04T09:44:52.8061422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8061499Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8061729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8061803Z layer_outputs = layer_module( 2025-12-04T09:44:52.8062013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8062089Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8062356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8062429Z return func(*args, **kwargs) 2025-12-04T09:44:52.8062708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8062803Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8063051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8063127Z return func(*args, **kwargs) 2025-12-04T09:44:52.8063371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8063455Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8063716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8063810Z return func(*args, **kwargs) 2025-12-04T09:44:52.8064078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8064162Z value_states = self.v(current_states) 2025-12-04T09:44:52.8064166Z 2025-12-04T09:44:52.8064259Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8064348Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8064453Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8064656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8064727Z res = mod(**inputs) 2025-12-04T09:44:52.8064982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8065067Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8065336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8065413Z layer_outputs = layer_module( 2025-12-04T09:44:52.8065671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8065752Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8066010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8066078Z return func(*args, **kwargs) 2025-12-04T09:44:52.8066324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8066413Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8066662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8066733Z return func(*args, **kwargs) 2025-12-04T09:44:52.8066990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8067076Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8067335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8067405Z return func(*args, **kwargs) 2025-12-04T09:44:52.8067655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8067739Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8067743Z 2025-12-04T09:44:52.8067848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8068054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8068125Z res = mod(**inputs) 2025-12-04T09:44:52.8068402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8068489Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8068736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8068840Z layer_outputs = layer_module( 2025-12-04T09:44:52.8069087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8069165Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8069413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8069482Z return func(*args, **kwargs) 2025-12-04T09:44:52.8069724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8069812Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8070088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8070156Z return func(*args, **kwargs) 2025-12-04T09:44:52.8070404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 529, in forward 2025-12-04T09:44:52.8070537Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:44:52.8070541Z 2025-12-04T09:44:52.8070650Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8070845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8070908Z res = mod(**inputs) 2025-12-04T09:44:52.8071154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8071228Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8071475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8071557Z layer_outputs = layer_module( 2025-12-04T09:44:52.8071778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8071876Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8072116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8072183Z return func(*args, **kwargs) 2025-12-04T09:44:52.8072426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8072517Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8072755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8072871Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8073116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.8073227Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.8073231Z 2025-12-04T09:44:52.8073333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8073535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8073606Z res = mod(**inputs) 2025-12-04T09:44:52.8073860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8073939Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8074174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8074244Z layer_outputs = layer_module( 2025-12-04T09:44:52.8074490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8074570Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8074836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8074924Z return func(*args, **kwargs) 2025-12-04T09:44:52.8075168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8075267Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8075510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8075628Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8075875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.8075979Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.8075983Z 2025-12-04T09:44:52.8076094Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8076307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8076374Z res = mod(**inputs) 2025-12-04T09:44:52.8076648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8076726Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8076990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8077072Z layer_outputs = layer_module( 2025-12-04T09:44:52.8077313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8077403Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8077668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8077742Z return func(*args, **kwargs) 2025-12-04T09:44:52.8078008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8078104Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8078369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8078492Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8078760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.8078862Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.8078865Z 2025-12-04T09:44:52.8078974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8079188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8079263Z res = mod(**inputs) 2025-12-04T09:44:52.8079524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8079608Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8079867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8079941Z layer_outputs = layer_module( 2025-12-04T09:44:52.8080190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8080270Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8080631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8080752Z return func(*args, **kwargs) 2025-12-04T09:44:52.8081028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8081157Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8081456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8081585Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8081870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.8081966Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.8081970Z 2025-12-04T09:44:52.8082084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8082282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8082348Z res = mod(**inputs) 2025-12-04T09:44:52.8082616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8082691Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8082926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8083004Z layer_outputs = layer_module( 2025-12-04T09:44:52.8083226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8083311Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8083558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8083628Z return func(*args, **kwargs) 2025-12-04T09:44:52.8083886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8083968Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8084209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8084278Z return func(*args, **kwargs) 2025-12-04T09:44:52.8084517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8084607Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8084857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8084924Z return func(*args, **kwargs) 2025-12-04T09:44:52.8085168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8085244Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8085247Z 2025-12-04T09:44:52.8085359Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8085553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8085617Z res = mod(**inputs) 2025-12-04T09:44:52.8085864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8085938Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8086183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8086253Z layer_outputs = layer_module( 2025-12-04T09:44:52.8086470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8086552Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8086793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8086894Z return func(*args, **kwargs) 2025-12-04T09:44:52.8087150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8087250Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8087534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8087603Z return func(*args, **kwargs) 2025-12-04T09:44:52.8087841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8087931Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8088171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8088238Z return func(*args, **kwargs) 2025-12-04T09:44:52.8088536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8088631Z key_states = self.k(current_states) 2025-12-04T09:44:52.8088635Z 2025-12-04T09:44:52.8088745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8088941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8089006Z res = mod(**inputs) 2025-12-04T09:44:52.8089256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8089329Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8089580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8089650Z layer_outputs = layer_module( 2025-12-04T09:44:52.8089870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8089959Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8090196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8090264Z return func(*args, **kwargs) 2025-12-04T09:44:52.8090510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8090590Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8090832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8090897Z return func(*args, **kwargs) 2025-12-04T09:44:52.8091136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8091225Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8091461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8091528Z return func(*args, **kwargs) 2025-12-04T09:44:52.8091771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8091850Z value_states = self.v(current_states) 2025-12-04T09:44:52.8091854Z 2025-12-04T09:44:52.8091940Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8092019Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8092121Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8092325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8092388Z res = mod(**inputs) 2025-12-04T09:44:52.8092628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8092706Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8092963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8093043Z layer_outputs = layer_module( 2025-12-04T09:44:52.8093280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8093375Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8093621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8093689Z return func(*args, **kwargs) 2025-12-04T09:44:52.8093933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8094013Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8094251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8094352Z return func(*args, **kwargs) 2025-12-04T09:44:52.8094592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8094676Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8094927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8094996Z return func(*args, **kwargs) 2025-12-04T09:44:52.8095244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8095319Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8095322Z 2025-12-04T09:44:52.8095426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8095632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8095696Z res = mod(**inputs) 2025-12-04T09:44:52.8095947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8096029Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8096283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8096360Z layer_outputs = layer_module( 2025-12-04T09:44:52.8096575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8096651Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8096891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8096957Z return func(*args, **kwargs) 2025-12-04T09:44:52.8097197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8097280Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8097518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8097595Z return func(*args, **kwargs) 2025-12-04T09:44:52.8097835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8097921Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8098168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8098236Z return func(*args, **kwargs) 2025-12-04T09:44:52.8098482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8098561Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8098564Z 2025-12-04T09:44:52.8098669Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8098896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8098964Z res = mod(**inputs) 2025-12-04T09:44:52.8099229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8099330Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8099574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8099651Z layer_outputs = layer_module( 2025-12-04T09:44:52.8099875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8099955Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8100208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8100303Z return func(*args, **kwargs) 2025-12-04T09:44:52.8100557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8100639Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8100892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8100966Z return func(*args, **kwargs) 2025-12-04T09:44:52.8101215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8101301Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8101555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8101626Z return func(*args, **kwargs) 2025-12-04T09:44:52.8101884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8101962Z key_states = self.k(current_states) 2025-12-04T09:44:52.8101966Z 2025-12-04T09:44:52.8102070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8102279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8102344Z res = mod(**inputs) 2025-12-04T09:44:52.8102592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8102674Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8102921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8102999Z layer_outputs = layer_module( 2025-12-04T09:44:52.8103227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8103306Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8103563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8103632Z return func(*args, **kwargs) 2025-12-04T09:44:52.8103885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8103966Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8104211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8104286Z return func(*args, **kwargs) 2025-12-04T09:44:52.8104533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8104616Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8104890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8104964Z return func(*args, **kwargs) 2025-12-04T09:44:52.8105263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8105364Z value_states = self.v(current_states) 2025-12-04T09:44:52.8105368Z 2025-12-04T09:44:52.8105469Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8105560Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8105665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8105867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8105939Z res = mod(**inputs) 2025-12-04T09:44:52.8106188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8106269Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8106519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8106624Z layer_outputs = layer_module( 2025-12-04T09:44:52.8106856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8106934Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8107183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8107252Z return func(*args, **kwargs) 2025-12-04T09:44:52.8107495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8107583Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8107824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8107894Z return func(*args, **kwargs) 2025-12-04T09:44:52.8108141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8108236Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8108479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8108544Z return func(*args, **kwargs) 2025-12-04T09:44:52.8108780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8108861Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8108864Z 2025-12-04T09:44:52.8108964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8109159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8109229Z res = mod(**inputs) 2025-12-04T09:44:52.8109469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8109548Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8109784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8109853Z layer_outputs = layer_module( 2025-12-04T09:44:52.8110073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8110152Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8110394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8110461Z return func(*args, **kwargs) 2025-12-04T09:44:52.8110693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8110812Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8111050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8111185Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8111443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.8111543Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.8111547Z 2025-12-04T09:44:52.8111655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8111850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8111913Z res = mod(**inputs) 2025-12-04T09:44:52.8112156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8112229Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8112484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8112562Z layer_outputs = layer_module( 2025-12-04T09:44:52.8112781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8112866Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8113102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8113170Z return func(*args, **kwargs) 2025-12-04T09:44:52.8113409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8113497Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8113738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8113857Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8114090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.8114178Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.8114181Z 2025-12-04T09:44:52.8114284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8114477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8114549Z res = mod(**inputs) 2025-12-04T09:44:52.8114788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8114867Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8115111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8115181Z layer_outputs = layer_module( 2025-12-04T09:44:52.8115400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8115477Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8115715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8115781Z return func(*args, **kwargs) 2025-12-04T09:44:52.8116007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8116097Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8116323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8116432Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8116683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.8116771Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.8116775Z 2025-12-04T09:44:52.8116905Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8117100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8117183Z res = mod(**inputs) 2025-12-04T09:44:52.8117428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8117498Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8117739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8117817Z layer_outputs = layer_module( 2025-12-04T09:44:52.8118035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8118141Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8118376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8118445Z return func(*args, **kwargs) 2025-12-04T09:44:52.8118689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8118776Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8119023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8119137Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8119377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.8119465Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.8119468Z 2025-12-04T09:44:52.8119577Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8119777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8119852Z res = mod(**inputs) 2025-12-04T09:44:52.8120099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8120179Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8120493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8120571Z layer_outputs = layer_module( 2025-12-04T09:44:52.8120805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8120885Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8121164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8121243Z return func(*args, **kwargs) 2025-12-04T09:44:52.8121506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8121615Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8121877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 218, in forward 2025-12-04T09:44:52.8122019Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-12-04T09:44:52.8122023Z 2025-12-04T09:44:52.8122156Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8122368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8122445Z res = mod(**inputs) 2025-12-04T09:44:52.8122704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8122816Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8123063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8123151Z layer_outputs = layer_module( 2025-12-04T09:44:52.8123393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8123480Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8123718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8123791Z return func(*args, **kwargs) 2025-12-04T09:44:52.8124026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8124107Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8124348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8124435Z return func(*args, **kwargs) 2025-12-04T09:44:52.8124682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8124764Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8125009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8125083Z return func(*args, **kwargs) 2025-12-04T09:44:52.8125322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8125399Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8125403Z 2025-12-04T09:44:52.8125513Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8125713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8125785Z res = mod(**inputs) 2025-12-04T09:44:52.8126029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8126103Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8126355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8126424Z layer_outputs = layer_module( 2025-12-04T09:44:52.8126646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8126732Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8126975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8127051Z return func(*args, **kwargs) 2025-12-04T09:44:52.8127295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8127375Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8127622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8127690Z return func(*args, **kwargs) 2025-12-04T09:44:52.8127939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8128021Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8128261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8128338Z return func(*args, **kwargs) 2025-12-04T09:44:52.8128579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8128655Z key_states = self.k(current_states) 2025-12-04T09:44:52.8128667Z 2025-12-04T09:44:52.8128791Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8128989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8129081Z res = mod(**inputs) 2025-12-04T09:44:52.8129369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8129445Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8129697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8129767Z layer_outputs = layer_module( 2025-12-04T09:44:52.8129995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8130075Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8130316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8130409Z return func(*args, **kwargs) 2025-12-04T09:44:52.8130642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8130723Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8130968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8131035Z return func(*args, **kwargs) 2025-12-04T09:44:52.8131275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8131355Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8131589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8131665Z return func(*args, **kwargs) 2025-12-04T09:44:52.8131911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8131989Z value_states = self.v(current_states) 2025-12-04T09:44:52.8132002Z 2025-12-04T09:44:52.8132086Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8132165Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8132278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8132479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8132544Z res = mod(**inputs) 2025-12-04T09:44:52.8132797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8132872Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8133126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8133207Z layer_outputs = layer_module( 2025-12-04T09:44:52.8133426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8133515Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8133755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8133823Z return func(*args, **kwargs) 2025-12-04T09:44:52.8134067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8134146Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8134392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8134461Z return func(*args, **kwargs) 2025-12-04T09:44:52.8134725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8134816Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8135060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8135149Z return func(*args, **kwargs) 2025-12-04T09:44:52.8135420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8135501Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8135505Z 2025-12-04T09:44:52.8135617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8135820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8135885Z res = mod(**inputs) 2025-12-04T09:44:52.8136140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8136228Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8136488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8136566Z layer_outputs = layer_module( 2025-12-04T09:44:52.8136784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8136868Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8137108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8137175Z return func(*args, **kwargs) 2025-12-04T09:44:52.8137416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8137497Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8137747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8137816Z return func(*args, **kwargs) 2025-12-04T09:44:52.8138054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8138149Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8138391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8138459Z return func(*args, **kwargs) 2025-12-04T09:44:52.8138706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8138784Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8138787Z 2025-12-04T09:44:52.8138896Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8139094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8139162Z res = mod(**inputs) 2025-12-04T09:44:52.8139411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8139485Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8139732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8139809Z layer_outputs = layer_module( 2025-12-04T09:44:52.8140033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8140119Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8140360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8140438Z return func(*args, **kwargs) 2025-12-04T09:44:52.8140698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8140779Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8141025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8141112Z return func(*args, **kwargs) 2025-12-04T09:44:52.8141369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8141463Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8141712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8141783Z return func(*args, **kwargs) 2025-12-04T09:44:52.8142040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8142118Z key_states = self.k(current_states) 2025-12-04T09:44:52.8142122Z 2025-12-04T09:44:52.8142257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8142456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8142524Z res = mod(**inputs) 2025-12-04T09:44:52.8142779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8142853Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8143098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8143179Z layer_outputs = layer_module( 2025-12-04T09:44:52.8143402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8143487Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8143733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8143804Z return func(*args, **kwargs) 2025-12-04T09:44:52.8144055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8144137Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8144389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8144458Z return func(*args, **kwargs) 2025-12-04T09:44:52.8144698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8144793Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8145039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8145108Z return func(*args, **kwargs) 2025-12-04T09:44:52.8145370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8145447Z value_states = self.v(current_states) 2025-12-04T09:44:52.8145452Z 2025-12-04T09:44:52.8145539Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8145617Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8145721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8145923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8145987Z res = mod(**inputs) 2025-12-04T09:44:52.8146234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8146315Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8146562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8146640Z layer_outputs = layer_module( 2025-12-04T09:44:52.8146892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8146992Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8147359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8147472Z return func(*args, **kwargs) 2025-12-04T09:44:52.8147720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8147811Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8148059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8148137Z return func(*args, **kwargs) 2025-12-04T09:44:52.8148382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8148496Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8148744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8148817Z return func(*args, **kwargs) 2025-12-04T09:44:52.8149065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8149143Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8149147Z 2025-12-04T09:44:52.8149253Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8149464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8149530Z res = mod(**inputs) 2025-12-04T09:44:52.8149775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8149860Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8150104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8150188Z layer_outputs = layer_module( 2025-12-04T09:44:52.8150411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8150489Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8150741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8150810Z return func(*args, **kwargs) 2025-12-04T09:44:52.8151062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8151156Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8151399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8151527Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8151767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.8151868Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.8151871Z 2025-12-04T09:44:52.8151985Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8152193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8152264Z res = mod(**inputs) 2025-12-04T09:44:52.8152508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8152583Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8152829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8152934Z layer_outputs = layer_module( 2025-12-04T09:44:52.8153157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8153279Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8153542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8153621Z return func(*args, **kwargs) 2025-12-04T09:44:52.8153860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8153953Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8154200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8154321Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8154568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.8154669Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.8154674Z 2025-12-04T09:44:52.8154780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8154990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8155057Z res = mod(**inputs) 2025-12-04T09:44:52.8155301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8155385Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8155634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8155719Z layer_outputs = layer_module( 2025-12-04T09:44:52.8155946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8156030Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8156282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8156354Z return func(*args, **kwargs) 2025-12-04T09:44:52.8156601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8156694Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8156935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8157059Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8157300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.8157387Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.8157393Z 2025-12-04T09:44:52.8157507Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8157707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8157780Z res = mod(**inputs) 2025-12-04T09:44:52.8158028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8158101Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8158352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8158424Z layer_outputs = layer_module( 2025-12-04T09:44:52.8158649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8158737Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8159004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8159086Z return func(*args, **kwargs) 2025-12-04T09:44:52.8159328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8159445Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8159725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8159847Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8160106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.8160193Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.8160197Z 2025-12-04T09:44:52.8160362Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8160607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8160703Z res = mod(**inputs) 2025-12-04T09:44:52.8160973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8161062Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8161329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8161414Z layer_outputs = layer_module( 2025-12-04T09:44:52.8161662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8161742Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8161995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8162066Z return func(*args, **kwargs) 2025-12-04T09:44:52.8162339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8162432Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8162720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8162803Z return func(*args, **kwargs) 2025-12-04T09:44:52.8163075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8163167Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8163450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8163525Z return func(*args, **kwargs) 2025-12-04T09:44:52.8163800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8163884Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8163892Z 2025-12-04T09:44:52.8164008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8164236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8164307Z res = mod(**inputs) 2025-12-04T09:44:52.8164589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8164675Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8164952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8165038Z layer_outputs = layer_module( 2025-12-04T09:44:52.8165282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8165368Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8165668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8165746Z return func(*args, **kwargs) 2025-12-04T09:44:52.8166068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8166180Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8166471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8166557Z return func(*args, **kwargs) 2025-12-04T09:44:52.8166820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8166911Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8167192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8167268Z return func(*args, **kwargs) 2025-12-04T09:44:52.8167572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8167658Z key_states = self.k(current_states) 2025-12-04T09:44:52.8167662Z 2025-12-04T09:44:52.8167775Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8168003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8168074Z res = mod(**inputs) 2025-12-04T09:44:52.8168350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8168436Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8168712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8168796Z layer_outputs = layer_module( 2025-12-04T09:44:52.8169043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8169131Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8169422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8169497Z return func(*args, **kwargs) 2025-12-04T09:44:52.8169779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8169867Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8170140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8170223Z return func(*args, **kwargs) 2025-12-04T09:44:52.8170499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8170585Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8170871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8170944Z return func(*args, **kwargs) 2025-12-04T09:44:52.8171213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8171296Z value_states = self.v(current_states) 2025-12-04T09:44:52.8171300Z 2025-12-04T09:44:52.8171386Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8171480Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8171590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8171810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8171886Z res = mod(**inputs) 2025-12-04T09:44:52.8172158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8172271Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8172533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8172631Z layer_outputs = layer_module( 2025-12-04T09:44:52.8172893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8172981Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8173243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8173323Z return func(*args, **kwargs) 2025-12-04T09:44:52.8173580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8173673Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8173931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8174027Z return func(*args, **kwargs) 2025-12-04T09:44:52.8174294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8174385Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8174656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8174728Z return func(*args, **kwargs) 2025-12-04T09:44:52.8174985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8175076Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8175079Z 2025-12-04T09:44:52.8175195Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8175409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8175488Z res = mod(**inputs) 2025-12-04T09:44:52.8175752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8175842Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8176103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8176181Z layer_outputs = layer_module( 2025-12-04T09:44:52.8176430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8176515Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8176777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8176858Z return func(*args, **kwargs) 2025-12-04T09:44:52.8177121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8177218Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8177482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8177556Z return func(*args, **kwargs) 2025-12-04T09:44:52.8177822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T09:44:52.8177967Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:44:52.8177971Z 2025-12-04T09:44:52.8178088Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8178303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8178373Z res = mod(**inputs) 2025-12-04T09:44:52.8178644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8178756Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8179018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8179123Z layer_outputs = layer_module( 2025-12-04T09:44:52.8179380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8179474Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8179737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8179811Z return func(*args, **kwargs) 2025-12-04T09:44:52.8180079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8180165Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8180435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8180532Z return func(*args, **kwargs) 2025-12-04T09:44:52.8180786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8180886Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8181144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8181218Z return func(*args, **kwargs) 2025-12-04T09:44:52.8181482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8181566Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8181570Z 2025-12-04T09:44:52.8181686Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8181900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8181972Z res = mod(**inputs) 2025-12-04T09:44:52.8182241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8182322Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8182585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8182668Z layer_outputs = layer_module( 2025-12-04T09:44:52.8182906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8182998Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8183252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8183324Z return func(*args, **kwargs) 2025-12-04T09:44:52.8183586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8183669Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8183916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8183987Z return func(*args, **kwargs) 2025-12-04T09:44:52.8184230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8184322Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8184563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8184632Z return func(*args, **kwargs) 2025-12-04T09:44:52.8184878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8184956Z key_states = self.k(current_states) 2025-12-04T09:44:52.8184980Z 2025-12-04T09:44:52.8185092Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8185292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8185376Z res = mod(**inputs) 2025-12-04T09:44:52.8185648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8185722Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8185974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8186056Z layer_outputs = layer_module( 2025-12-04T09:44:52.8186293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8186384Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8186652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8186743Z return func(*args, **kwargs) 2025-12-04T09:44:52.8186992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8187074Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8187323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8187392Z return func(*args, **kwargs) 2025-12-04T09:44:52.8187631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8187724Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8187965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8188034Z return func(*args, **kwargs) 2025-12-04T09:44:52.8188285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8188363Z value_states = self.v(current_states) 2025-12-04T09:44:52.8188368Z 2025-12-04T09:44:52.8188455Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8188533Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8188639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8188844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8188910Z res = mod(**inputs) 2025-12-04T09:44:52.8189161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8189244Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8189488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8189568Z layer_outputs = layer_module( 2025-12-04T09:44:52.8189790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8189871Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8190122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8190191Z return func(*args, **kwargs) 2025-12-04T09:44:52.8190433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8190520Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8190766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8190842Z return func(*args, **kwargs) 2025-12-04T09:44:52.8191105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8191193Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8191441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8191529Z return func(*args, **kwargs) 2025-12-04T09:44:52.8191800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8191880Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8191884Z 2025-12-04T09:44:52.8191988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8192205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8192271Z res = mod(**inputs) 2025-12-04T09:44:52.8192519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8192624Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8192877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8192957Z layer_outputs = layer_module( 2025-12-04T09:44:52.8193195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8193273Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8193532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8193599Z return func(*args, **kwargs) 2025-12-04T09:44:52.8193850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8193950Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8194203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8194331Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8194584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.8194684Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.8194690Z 2025-12-04T09:44:52.8194803Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8195010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8195080Z res = mod(**inputs) 2025-12-04T09:44:52.8195335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8195409Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8195672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8195746Z layer_outputs = layer_module( 2025-12-04T09:44:52.8195979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8196068Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8196322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8196398Z return func(*args, **kwargs) 2025-12-04T09:44:52.8196649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8196740Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8196998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8197116Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8197390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.8197473Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.8197499Z 2025-12-04T09:44:52.8197606Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8197834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8197900Z res = mod(**inputs) 2025-12-04T09:44:52.8198148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8198228Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8198471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8198549Z layer_outputs = layer_module( 2025-12-04T09:44:52.8198776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8198893Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8199142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8199214Z return func(*args, **kwargs) 2025-12-04T09:44:52.8199461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8199552Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8199789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8199911Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8200162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.8200260Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.8200266Z 2025-12-04T09:44:52.8200454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8200684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8200769Z res = mod(**inputs) 2025-12-04T09:44:52.8201046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8201126Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8201399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8201477Z layer_outputs = layer_module( 2025-12-04T09:44:52.8201721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8201809Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8202057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8202140Z return func(*args, **kwargs) 2025-12-04T09:44:52.8202408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8202507Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8202786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8202912Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8203187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.8203275Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.8203280Z 2025-12-04T09:44:52.8203395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8203655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8203731Z res = mod(**inputs) 2025-12-04T09:44:52.8204012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8204122Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8204426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8204513Z layer_outputs = layer_module( 2025-12-04T09:44:52.8204755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8204845Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8205121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8205197Z return func(*args, **kwargs) 2025-12-04T09:44:52.8205500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8205591Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8205878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8205961Z return func(*args, **kwargs) 2025-12-04T09:44:52.8206228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8206320Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8206598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8206672Z return func(*args, **kwargs) 2025-12-04T09:44:52.8206954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8207042Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8207046Z 2025-12-04T09:44:52.8207159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8207385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8207456Z res = mod(**inputs) 2025-12-04T09:44:52.8207734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8207822Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8208097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8208182Z layer_outputs = layer_module( 2025-12-04T09:44:52.8208427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8208513Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8208799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8208882Z return func(*args, **kwargs) 2025-12-04T09:44:52.8209126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8209207Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8209445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8209520Z return func(*args, **kwargs) 2025-12-04T09:44:52.8209755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8209837Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8210080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8210172Z return func(*args, **kwargs) 2025-12-04T09:44:52.8210422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8210517Z key_states = self.k(current_states) 2025-12-04T09:44:52.8210521Z 2025-12-04T09:44:52.8210625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8210846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8210913Z res = mod(**inputs) 2025-12-04T09:44:52.8211153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8211233Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8211472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8211550Z layer_outputs = layer_module( 2025-12-04T09:44:52.8211788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8211866Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8212110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8212178Z return func(*args, **kwargs) 2025-12-04T09:44:52.8212418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8212498Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8212736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8212811Z return func(*args, **kwargs) 2025-12-04T09:44:52.8213042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8213124Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8213368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8213436Z return func(*args, **kwargs) 2025-12-04T09:44:52.8213686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8213766Z value_states = self.v(current_states) 2025-12-04T09:44:52.8213769Z 2025-12-04T09:44:52.8213847Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8213934Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8214038Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8214245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8214314Z res = mod(**inputs) 2025-12-04T09:44:52.8214551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8214630Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8214865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8214936Z layer_outputs = layer_module( 2025-12-04T09:44:52.8215164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8215243Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8215477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8215550Z return func(*args, **kwargs) 2025-12-04T09:44:52.8215784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:44:52.8215872Z self_attention_outputs = self.layer[0]( 2025-12-04T09:44:52.8216129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8216201Z return func(*args, **kwargs) 2025-12-04T09:44:52.8216466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:44:52.8216565Z attention_output = self.SelfAttention( 2025-12-04T09:44:52.8216812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8216879Z return func(*args, **kwargs) 2025-12-04T09:44:52.8217115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8217196Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8217199Z 2025-12-04T09:44:52.8217301Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8217497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8217589Z res = mod(**inputs) 2025-12-04T09:44:52.8217829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8217909Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8218148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8218218Z layer_outputs = layer_module( 2025-12-04T09:44:52.8218442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8218519Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8218754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8218829Z return func(*args, **kwargs) 2025-12-04T09:44:52.8219065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8219152Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8219388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8219454Z return func(*args, **kwargs) 2025-12-04T09:44:52.8219697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8219780Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8220026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8220095Z return func(*args, **kwargs) 2025-12-04T09:44:52.8220330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:44:52.8220414Z query_states = self.q(hidden_states) 2025-12-04T09:44:52.8220419Z 2025-12-04T09:44:52.8220522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8220714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8220786Z res = mod(**inputs) 2025-12-04T09:44:52.8221023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8221104Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8221342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8221411Z layer_outputs = layer_module( 2025-12-04T09:44:52.8221638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8221714Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8221970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8222049Z return func(*args, **kwargs) 2025-12-04T09:44:52.8222308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8222396Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8222657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8222727Z return func(*args, **kwargs) 2025-12-04T09:44:52.8222975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8223061Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8223307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8223387Z return func(*args, **kwargs) 2025-12-04T09:44:52.8223637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:44:52.8223718Z key_states = self.k(current_states) 2025-12-04T09:44:52.8223722Z 2025-12-04T09:44:52.8223822Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8224016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8224088Z res = mod(**inputs) 2025-12-04T09:44:52.8224329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8224407Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8224643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8224713Z layer_outputs = layer_module( 2025-12-04T09:44:52.8224944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8225024Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8225267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8225343Z return func(*args, **kwargs) 2025-12-04T09:44:52.8225583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8225671Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8225914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8225986Z return func(*args, **kwargs) 2025-12-04T09:44:52.8226251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8226341Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8226616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8226688Z return func(*args, **kwargs) 2025-12-04T09:44:52.8226930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:44:52.8227017Z value_states = self.v(current_states) 2025-12-04T09:44:52.8227020Z 2025-12-04T09:44:52.8227100Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8227180Z cudagraph partition due to non gpu ops 2025-12-04T09:44:52.8227292Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8227491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8227561Z res = mod(**inputs) 2025-12-04T09:44:52.8227829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8227904Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8228154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8228245Z layer_outputs = layer_module( 2025-12-04T09:44:52.8228486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8228573Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8228816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8228892Z return func(*args, **kwargs) 2025-12-04T09:44:52.8229134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8229214Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8229461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8229550Z return func(*args, **kwargs) 2025-12-04T09:44:52.8229797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:44:52.8229880Z attention_output = self.EncDecAttention( 2025-12-04T09:44:52.8230123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8230198Z return func(*args, **kwargs) 2025-12-04T09:44:52.8230441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:44:52.8230518Z attn_output = self.o(attn_output) 2025-12-04T09:44:52.8230521Z 2025-12-04T09:44:52.8230634Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8230836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8230910Z res = mod(**inputs) 2025-12-04T09:44:52.8231156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8231230Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8231485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8231557Z layer_outputs = layer_module( 2025-12-04T09:44:52.8231781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8231867Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8232112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8232187Z return func(*args, **kwargs) 2025-12-04T09:44:52.8232439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:44:52.8232525Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:44:52.8232792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8232867Z return func(*args, **kwargs) 2025-12-04T09:44:52.8233134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 529, in forward 2025-12-04T09:44:52.8233270Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:44:52.8233274Z 2025-12-04T09:44:52.8233378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8233588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8233654Z res = mod(**inputs) 2025-12-04T09:44:52.8233921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8234008Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8234250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8234352Z layer_outputs = layer_module( 2025-12-04T09:44:52.8234594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8234675Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8234926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8234997Z return func(*args, **kwargs) 2025-12-04T09:44:52.8235239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8235340Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8235584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8235731Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8235973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:44:52.8236075Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:44:52.8236078Z 2025-12-04T09:44:52.8236190Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8236391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8236463Z res = mod(**inputs) 2025-12-04T09:44:52.8236711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8236790Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8237057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8237134Z layer_outputs = layer_module( 2025-12-04T09:44:52.8237371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8237465Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8237725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8237807Z return func(*args, **kwargs) 2025-12-04T09:44:52.8238059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8238158Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8238419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8238544Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8238810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:44:52.8238897Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:44:52.8238900Z 2025-12-04T09:44:52.8239008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8239228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8239297Z res = mod(**inputs) 2025-12-04T09:44:52.8239558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8239643Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8239902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8239986Z layer_outputs = layer_module( 2025-12-04T09:44:52.8240243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8240399Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8240707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8240782Z return func(*args, **kwargs) 2025-12-04T09:44:52.8241059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8241168Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8241427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8241559Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8241818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:44:52.8241938Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:44:52.8241942Z 2025-12-04T09:44:52.8242061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8242277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8242354Z res = mod(**inputs) 2025-12-04T09:44:52.8242616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:44:52.8242699Z decoder_outputs = self.decoder( 2025-12-04T09:44:52.8242967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:44:52.8243048Z layer_outputs = layer_module( 2025-12-04T09:44:52.8243287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:52.8243383Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:52.8243647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:52.8243732Z return func(*args, **kwargs) 2025-12-04T09:44:52.8243988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:44:52.8244088Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:44:52.8244352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:44:52.8244477Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:44:52.8244745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:44:52.8244834Z hidden_states = self.wo(hidden_states) 2025-12-04T09:44:52.8244838Z 2025-12-04T09:44:52.8244950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8245174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8245246Z res = mod(**inputs) 2025-12-04T09:44:52.8245511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1816, in forward 2025-12-04T09:44:52.8245618Z lm_logits = self.lm_head(sequence_output) 2025-12-04T09:44:52.8245622Z 2025-12-04T09:44:52.8245733Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:52.8245971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:52.8246043Z res = mod(**inputs) 2025-12-04T09:44:52.8246328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-12-04T09:44:52.8246493Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-12-04T09:44:52.8246497Z 2025-12-04T09:45:03.1048897Z Compilation time (from dynamo_timed): 21.073274682 2025-12-04T09:45:03.1283951Z pass 2025-12-04T09:45:03.1284466Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:03.1285580Z TIMING: _recursive_pre_grad_passes:0.01477 _recursive_joint_graph_passes:0.79978 _recursive_post_grad_passes:0.08555 async_compile.wait:0.75068 code_gen:9.88226 inductor_compile:12.09476 backend_compile:17.18252 gc:0.00018 entire_frame_compile:21.07327 total_wall_time:21.07327 2025-12-04T09:45:03.1286513Z STATS: call_* op count: 1189 | FakeTensorMode.__torch_dispatch__:16428 | FakeTensor.__torch_dispatch__:6387 | ProxyTorchDispatchMode.__torch_dispatch__:4865 2025-12-04T09:45:03.1287087Z Dynamo produced 1 graphs covering 1189 ops with 0 graph breaks (0 unique) 2025-12-04T09:45:05.6700348Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:45:05.6701940Z import pynvml # type: ignore[import] 2025-12-04T09:45:09.2305478Z 2025-12-04T09:45:09.2441372Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T09:45:09.2442143Z WARNING:transformers.models.megatron_bert.modeling_megatron_bert:If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T09:45:12.0936901Z 2025-12-04T09:45:12.0937854Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:45:12.0959639Z cpu eval MegatronBertForCausalLM 2025-12-04T09:45:13.7118340Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:14.3375862Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:15.1940160Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:29.7471128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7472832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7473306Z res = mod(**inputs) 2025-12-04T09:45:29.7479784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7480486Z outputs = self.bert( 2025-12-04T09:45:29.7480961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7481429Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7481916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7482356Z layer_outputs = layer_module( 2025-12-04T09:45:29.7482738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7483168Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7483611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7484030Z return func(*args, **kwargs) 2025-12-04T09:45:29.7484483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7484932Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7485330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7485715Z return func(*args, **kwargs) 2025-12-04T09:45:29.7486422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7486865Z self_outputs = self.self( 2025-12-04T09:45:29.7487248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7487687Z return func(*args, **kwargs) 2025-12-04T09:45:29.7488153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.7488628Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.7488775Z 2025-12-04T09:45:29.7488898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7489272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7489612Z res = mod(**inputs) 2025-12-04T09:45:29.7490031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7490527Z outputs = self.bert( 2025-12-04T09:45:29.7490931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7491371Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7491806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7492232Z layer_outputs = layer_module( 2025-12-04T09:45:29.7492596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7492975Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7493373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7493756Z return func(*args, **kwargs) 2025-12-04T09:45:29.7494201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7494670Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7495090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7495488Z return func(*args, **kwargs) 2025-12-04T09:45:29.7495923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7496357Z self_outputs = self.self( 2025-12-04T09:45:29.7496721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7497115Z return func(*args, **kwargs) 2025-12-04T09:45:29.7497541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.7497980Z key_layer = self.key(current_states) 2025-12-04T09:45:29.7498125Z 2025-12-04T09:45:29.7498242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7498643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7498994Z res = mod(**inputs) 2025-12-04T09:45:29.7499432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7499879Z outputs = self.bert( 2025-12-04T09:45:29.7500307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7500766Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7501210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7501694Z layer_outputs = layer_module( 2025-12-04T09:45:29.7502079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7502497Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7502915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7503317Z return func(*args, **kwargs) 2025-12-04T09:45:29.7503764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7504226Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7504630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7505029Z return func(*args, **kwargs) 2025-12-04T09:45:29.7505473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7505937Z self_outputs = self.self( 2025-12-04T09:45:29.7506337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7506740Z return func(*args, **kwargs) 2025-12-04T09:45:29.7507170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.7507636Z value_layer = self.value(current_states) 2025-12-04T09:45:29.7507792Z 2025-12-04T09:45:29.7507886Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7508134Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7508394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7508791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7509147Z res = mod(**inputs) 2025-12-04T09:45:29.7509595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7510048Z outputs = self.bert( 2025-12-04T09:45:29.7510483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7510943Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7511392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7511854Z layer_outputs = layer_module( 2025-12-04T09:45:29.7512234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7512628Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7513034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7513443Z return func(*args, **kwargs) 2025-12-04T09:45:29.7513891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7514370Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7514779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7515183Z return func(*args, **kwargs) 2025-12-04T09:45:29.7515628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.7516146Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.7516680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.7517187Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7517343Z 2025-12-04T09:45:29.7517465Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7517877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7518232Z res = mod(**inputs) 2025-12-04T09:45:29.7518710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7519175Z outputs = self.bert( 2025-12-04T09:45:29.7519611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7520079Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7520654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7521152Z layer_outputs = layer_module( 2025-12-04T09:45:29.7521554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7521951Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7522362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7522755Z return func(*args, **kwargs) 2025-12-04T09:45:29.7523195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7523667Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7524107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7524531Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7525019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7525536Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7526000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.7526464Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7526620Z 2025-12-04T09:45:29.7526736Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7527125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7527464Z res = mod(**inputs) 2025-12-04T09:45:29.7527891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7528366Z outputs = self.bert( 2025-12-04T09:45:29.7528798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7529257Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7529686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7530104Z layer_outputs = layer_module( 2025-12-04T09:45:29.7530460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7530834Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7531232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7531665Z return func(*args, **kwargs) 2025-12-04T09:45:29.7532083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7532553Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7532976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7533422Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7533942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7534477Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7534960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.7535462Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.7535852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.7536196Z return self.act(input) 2025-12-04T09:45:29.7536341Z 2025-12-04T09:45:29.7536449Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7536834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7537183Z res = mod(**inputs) 2025-12-04T09:45:29.7537608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7538060Z outputs = self.bert( 2025-12-04T09:45:29.7538495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7538942Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7539393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7539823Z layer_outputs = layer_module( 2025-12-04T09:45:29.7540182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7540551Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7540942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7541323Z return func(*args, **kwargs) 2025-12-04T09:45:29.7541742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7542182Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7542599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7542998Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7543455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.7543995Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.7544487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.7544936Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7545078Z 2025-12-04T09:45:29.7545188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7545557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7545887Z res = mod(**inputs) 2025-12-04T09:45:29.7546295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7546712Z outputs = self.bert( 2025-12-04T09:45:29.7547392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7547876Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7548306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7548766Z layer_outputs = layer_module( 2025-12-04T09:45:29.7549174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7549570Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7549972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7550360Z return func(*args, **kwargs) 2025-12-04T09:45:29.7550781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7551255Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7551671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7552053Z return func(*args, **kwargs) 2025-12-04T09:45:29.7552468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7552897Z self_outputs = self.self( 2025-12-04T09:45:29.7553263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7553640Z return func(*args, **kwargs) 2025-12-04T09:45:29.7554056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.7554488Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.7554637Z 2025-12-04T09:45:29.7554745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7555114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7555442Z res = mod(**inputs) 2025-12-04T09:45:29.7555838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7556266Z outputs = self.bert( 2025-12-04T09:45:29.7556669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7557097Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7557519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7557940Z layer_outputs = layer_module( 2025-12-04T09:45:29.7558296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7558676Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7559083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7559493Z return func(*args, **kwargs) 2025-12-04T09:45:29.7559941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7560474Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7560904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7561318Z return func(*args, **kwargs) 2025-12-04T09:45:29.7561770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7562220Z self_outputs = self.self( 2025-12-04T09:45:29.7562637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7563048Z return func(*args, **kwargs) 2025-12-04T09:45:29.7563503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.7563981Z key_layer = self.key(current_states) 2025-12-04T09:45:29.7564130Z 2025-12-04T09:45:29.7564251Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7564639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7564977Z res = mod(**inputs) 2025-12-04T09:45:29.7565403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7565852Z outputs = self.bert( 2025-12-04T09:45:29.7566273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7566743Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7567196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7567646Z layer_outputs = layer_module( 2025-12-04T09:45:29.7568019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7568412Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7568827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7569228Z return func(*args, **kwargs) 2025-12-04T09:45:29.7569663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7570129Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7570538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7570944Z return func(*args, **kwargs) 2025-12-04T09:45:29.7571376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7571796Z self_outputs = self.self( 2025-12-04T09:45:29.7572166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7572535Z return func(*args, **kwargs) 2025-12-04T09:45:29.7572953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.7573390Z value_layer = self.value(current_states) 2025-12-04T09:45:29.7573538Z 2025-12-04T09:45:29.7573628Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7573839Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7574077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7574439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7574757Z res = mod(**inputs) 2025-12-04T09:45:29.7575167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7575595Z outputs = self.bert( 2025-12-04T09:45:29.7576002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7576456Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7576915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7577390Z layer_outputs = layer_module( 2025-12-04T09:45:29.7577765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7578173Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7579404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7579807Z return func(*args, **kwargs) 2025-12-04T09:45:29.7580219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7580660Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7581058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7581434Z return func(*args, **kwargs) 2025-12-04T09:45:29.7581864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.7582384Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.7582878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.7583321Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7583472Z 2025-12-04T09:45:29.7583582Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7583957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7584290Z res = mod(**inputs) 2025-12-04T09:45:29.7585171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7585604Z outputs = self.bert( 2025-12-04T09:45:29.7586018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7586476Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7586928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7587394Z layer_outputs = layer_module( 2025-12-04T09:45:29.7587779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7588144Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7588531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7588913Z return func(*args, **kwargs) 2025-12-04T09:45:29.7589331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7589774Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7590191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7590604Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7591061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7591555Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7592019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.7592462Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7592602Z 2025-12-04T09:45:29.7592712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7593083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7593458Z res = mod(**inputs) 2025-12-04T09:45:29.7593900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7594353Z outputs = self.bert( 2025-12-04T09:45:29.7594798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7595259Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7595710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7596154Z layer_outputs = layer_module( 2025-12-04T09:45:29.7596536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7596936Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7597349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7597794Z return func(*args, **kwargs) 2025-12-04T09:45:29.7598246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7598738Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7599171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7599610Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7600123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7600787Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7601298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.7601809Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.7602226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.7602578Z return self.act(input) 2025-12-04T09:45:29.7602711Z 2025-12-04T09:45:29.7602825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7603219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7603572Z res = mod(**inputs) 2025-12-04T09:45:29.7603994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7604447Z outputs = self.bert( 2025-12-04T09:45:29.7604872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7605330Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7605768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7606218Z layer_outputs = layer_module( 2025-12-04T09:45:29.7606591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7606982Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7607393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7607794Z return func(*args, **kwargs) 2025-12-04T09:45:29.7608238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7608693Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7609154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7609585Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7610085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.7610652Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.7611190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.7611659Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7611808Z 2025-12-04T09:45:29.7611925Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7612325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7612675Z res = mod(**inputs) 2025-12-04T09:45:29.7613130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7613576Z outputs = self.bert( 2025-12-04T09:45:29.7614003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7614456Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7614902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7615328Z layer_outputs = layer_module( 2025-12-04T09:45:29.7615685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7616056Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7616438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7616830Z return func(*args, **kwargs) 2025-12-04T09:45:29.7617247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7617703Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7618145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7618547Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7618999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.7619515Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.7619997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.7620431Z return input_tensor + hidden_states 2025-12-04T09:45:29.7620565Z 2025-12-04T09:45:29.7620680Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7621042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7621374Z res = mod(**inputs) 2025-12-04T09:45:29.7621791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7622243Z outputs = self.bert( 2025-12-04T09:45:29.7622662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7623117Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7623539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7623981Z layer_outputs = layer_module( 2025-12-04T09:45:29.7624330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7624720Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7625130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7625503Z return func(*args, **kwargs) 2025-12-04T09:45:29.7625918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7626379Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7626793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7627163Z return func(*args, **kwargs) 2025-12-04T09:45:29.7627595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7628069Z self_outputs = self.self( 2025-12-04T09:45:29.7648819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7649245Z return func(*args, **kwargs) 2025-12-04T09:45:29.7649693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.7650150Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.7650305Z 2025-12-04T09:45:29.7650418Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7650795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7651127Z res = mod(**inputs) 2025-12-04T09:45:29.7651541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7651978Z outputs = self.bert( 2025-12-04T09:45:29.7652384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7652810Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7653234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7653662Z layer_outputs = layer_module( 2025-12-04T09:45:29.7654016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7654385Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7654777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7655157Z return func(*args, **kwargs) 2025-12-04T09:45:29.7655570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7656006Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7656389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7656767Z return func(*args, **kwargs) 2025-12-04T09:45:29.7657181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7657596Z self_outputs = self.self( 2025-12-04T09:45:29.7657957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7658338Z return func(*args, **kwargs) 2025-12-04T09:45:29.7658891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.7659332Z key_layer = self.key(current_states) 2025-12-04T09:45:29.7659482Z 2025-12-04T09:45:29.7659638Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7660026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7660390Z res = mod(**inputs) 2025-12-04T09:45:29.7660796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7661224Z outputs = self.bert( 2025-12-04T09:45:29.7661633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7662056Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7662487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7662961Z layer_outputs = layer_module( 2025-12-04T09:45:29.7663325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7663693Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7664085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7664463Z return func(*args, **kwargs) 2025-12-04T09:45:29.7664879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7665312Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7665701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7666076Z return func(*args, **kwargs) 2025-12-04T09:45:29.7666484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7666918Z self_outputs = self.self( 2025-12-04T09:45:29.7667290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7667671Z return func(*args, **kwargs) 2025-12-04T09:45:29.7668079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.7668515Z value_layer = self.value(current_states) 2025-12-04T09:45:29.7668653Z 2025-12-04T09:45:29.7668748Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7668968Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7669205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7669576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7669908Z res = mod(**inputs) 2025-12-04T09:45:29.7670308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7670738Z outputs = self.bert( 2025-12-04T09:45:29.7671145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7671570Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7671991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7672420Z layer_outputs = layer_module( 2025-12-04T09:45:29.7672781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7673183Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7673577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7673950Z return func(*args, **kwargs) 2025-12-04T09:45:29.7674378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7674815Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7675198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7675566Z return func(*args, **kwargs) 2025-12-04T09:45:29.7675963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.7676434Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.7676899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.7677344Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7677481Z 2025-12-04T09:45:29.7677586Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7677947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7678275Z res = mod(**inputs) 2025-12-04T09:45:29.7678679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7679094Z outputs = self.bert( 2025-12-04T09:45:29.7679504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7679956Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7680475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7680945Z layer_outputs = layer_module( 2025-12-04T09:45:29.7681352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7681753Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7682163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7682546Z return func(*args, **kwargs) 2025-12-04T09:45:29.7682977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7683408Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7683812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7684209Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7684667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7685150Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7685597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.7686029Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7686164Z 2025-12-04T09:45:29.7686275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7686626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7686948Z res = mod(**inputs) 2025-12-04T09:45:29.7687344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7687755Z outputs = self.bert( 2025-12-04T09:45:29.7688164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7688597Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7689029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7689439Z layer_outputs = layer_module( 2025-12-04T09:45:29.7689787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7690145Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7690527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7690887Z return func(*args, **kwargs) 2025-12-04T09:45:29.7691296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7691750Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7692156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7692548Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7692996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7693476Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7693921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.7694383Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.7694758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.7695095Z return self.act(input) 2025-12-04T09:45:29.7695204Z 2025-12-04T09:45:29.7695306Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7695670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7695988Z res = mod(**inputs) 2025-12-04T09:45:29.7696385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7696782Z outputs = self.bert( 2025-12-04T09:45:29.7697169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7697578Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7697982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7698399Z layer_outputs = layer_module( 2025-12-04T09:45:29.7698755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7699123Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7699491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7699866Z return func(*args, **kwargs) 2025-12-04T09:45:29.7700285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7700706Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7701093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7701476Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7701924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.7702411Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.7702893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.7703386Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7703521Z 2025-12-04T09:45:29.7703631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7703973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7704282Z res = mod(**inputs) 2025-12-04T09:45:29.7704668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7705070Z outputs = self.bert( 2025-12-04T09:45:29.7705447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7705873Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7706281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7706679Z layer_outputs = layer_module( 2025-12-04T09:45:29.7707019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7707372Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7707737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7708095Z return func(*args, **kwargs) 2025-12-04T09:45:29.7708489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7708907Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7709266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7709623Z return func(*args, **kwargs) 2025-12-04T09:45:29.7710020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7710417Z self_outputs = self.self( 2025-12-04T09:45:29.7710758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7711112Z return func(*args, **kwargs) 2025-12-04T09:45:29.7711499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.7711902Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.7712039Z 2025-12-04T09:45:29.7712143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7712485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7712794Z res = mod(**inputs) 2025-12-04T09:45:29.7713168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7713565Z outputs = self.bert( 2025-12-04T09:45:29.7713947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7714346Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7714739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7715138Z layer_outputs = layer_module( 2025-12-04T09:45:29.7715489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7715839Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7716211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7716590Z return func(*args, **kwargs) 2025-12-04T09:45:29.7717003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7717413Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7717782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7718140Z return func(*args, **kwargs) 2025-12-04T09:45:29.7718527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7718938Z self_outputs = self.self( 2025-12-04T09:45:29.7719315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7719681Z return func(*args, **kwargs) 2025-12-04T09:45:29.7720079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.7720623Z key_layer = self.key(current_states) 2025-12-04T09:45:29.7720761Z 2025-12-04T09:45:29.7720877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7721258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7721615Z res = mod(**inputs) 2025-12-04T09:45:29.7722082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7722565Z outputs = self.bert( 2025-12-04T09:45:29.7723017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7723475Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7723897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7724322Z layer_outputs = layer_module( 2025-12-04T09:45:29.7724669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7725037Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7725431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7725807Z return func(*args, **kwargs) 2025-12-04T09:45:29.7726228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7726675Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7727071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7727435Z return func(*args, **kwargs) 2025-12-04T09:45:29.7727848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7728264Z self_outputs = self.self( 2025-12-04T09:45:29.7728622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7728986Z return func(*args, **kwargs) 2025-12-04T09:45:29.7729392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.7729825Z value_layer = self.value(current_states) 2025-12-04T09:45:29.7729980Z 2025-12-04T09:45:29.7730063Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7730278Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7730532Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7730888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7731218Z res = mod(**inputs) 2025-12-04T09:45:29.7731616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7732031Z outputs = self.bert( 2025-12-04T09:45:29.7732423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7732838Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7733255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7733691Z layer_outputs = layer_module( 2025-12-04T09:45:29.7734033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7734395Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7734775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7735154Z return func(*args, **kwargs) 2025-12-04T09:45:29.7735585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7736047Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7736463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7736845Z return func(*args, **kwargs) 2025-12-04T09:45:29.7737270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.7737743Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.7738249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.7738690Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7738842Z 2025-12-04T09:45:29.7738953Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7739329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7739679Z res = mod(**inputs) 2025-12-04T09:45:29.7740093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7740555Z outputs = self.bert( 2025-12-04T09:45:29.7740982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7741417Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7741850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7742287Z layer_outputs = layer_module( 2025-12-04T09:45:29.7742655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7743043Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7743437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7743829Z return func(*args, **kwargs) 2025-12-04T09:45:29.7744268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7744726Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7745154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7745592Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7746075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7746584Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7747205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.7747662Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7747806Z 2025-12-04T09:45:29.7747916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7748292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7748673Z res = mod(**inputs) 2025-12-04T09:45:29.7749078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7749497Z outputs = self.bert( 2025-12-04T09:45:29.7749901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7750332Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7750749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7751177Z layer_outputs = layer_module( 2025-12-04T09:45:29.7751535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7751908Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7752293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7752674Z return func(*args, **kwargs) 2025-12-04T09:45:29.7753093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7753529Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7753941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7754343Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7754799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7755280Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7755743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.7756211Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.7756604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.7756948Z return self.act(input) 2025-12-04T09:45:29.7757072Z 2025-12-04T09:45:29.7757178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7757546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7757865Z res = mod(**inputs) 2025-12-04T09:45:29.7758290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7758738Z outputs = self.bert( 2025-12-04T09:45:29.7759193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7759644Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7760091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7760642Z layer_outputs = layer_module( 2025-12-04T09:45:29.7761051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7761437Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7761854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7762234Z return func(*args, **kwargs) 2025-12-04T09:45:29.7762644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7763095Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7763516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7763915Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7764357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.7764882Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.7765372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.7765814Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7765958Z 2025-12-04T09:45:29.7766068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7766448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7766786Z res = mod(**inputs) 2025-12-04T09:45:29.7767177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7767596Z outputs = self.bert( 2025-12-04T09:45:29.7768002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7768426Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7768847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7769297Z layer_outputs = layer_module( 2025-12-04T09:45:29.7769653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7770019Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7770402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7770815Z return func(*args, **kwargs) 2025-12-04T09:45:29.7771226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7771656Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7772054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7772442Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7772887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.7773385Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.7773891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.7774316Z return input_tensor + hidden_states 2025-12-04T09:45:29.7774467Z 2025-12-04T09:45:29.7774579Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7774937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7775280Z res = mod(**inputs) 2025-12-04T09:45:29.7775690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7776115Z outputs = self.bert( 2025-12-04T09:45:29.7776507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7776929Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7777366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7777788Z layer_outputs = layer_module( 2025-12-04T09:45:29.7778139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7778501Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7778892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7779271Z return func(*args, **kwargs) 2025-12-04T09:45:29.7779677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7780107Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7780480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7780844Z return func(*args, **kwargs) 2025-12-04T09:45:29.7781251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7781653Z self_outputs = self.self( 2025-12-04T09:45:29.7782003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7782364Z return func(*args, **kwargs) 2025-12-04T09:45:29.7782758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.7783180Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.7783312Z 2025-12-04T09:45:29.7783415Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7783771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7784090Z res = mod(**inputs) 2025-12-04T09:45:29.7784478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7784890Z outputs = self.bert( 2025-12-04T09:45:29.7785280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7785709Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7786129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7786538Z layer_outputs = layer_module( 2025-12-04T09:45:29.7786884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7787244Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7787623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7788018Z return func(*args, **kwargs) 2025-12-04T09:45:29.7788432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7788866Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7789274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7789641Z return func(*args, **kwargs) 2025-12-04T09:45:29.7790056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7790472Z self_outputs = self.self( 2025-12-04T09:45:29.7790843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7791221Z return func(*args, **kwargs) 2025-12-04T09:45:29.7791634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.7792087Z key_layer = self.key(current_states) 2025-12-04T09:45:29.7792232Z 2025-12-04T09:45:29.7792340Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7792706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7793030Z res = mod(**inputs) 2025-12-04T09:45:29.7793436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7793859Z outputs = self.bert( 2025-12-04T09:45:29.7794262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7794684Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7795114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7795544Z layer_outputs = layer_module( 2025-12-04T09:45:29.7795899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7796270Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7796665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7797069Z return func(*args, **kwargs) 2025-12-04T09:45:29.7797504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7797968Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7798382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7798779Z return func(*args, **kwargs) 2025-12-04T09:45:29.7799209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7799652Z self_outputs = self.self( 2025-12-04T09:45:29.7800038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7800523Z return func(*args, **kwargs) 2025-12-04T09:45:29.7800985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.7801465Z value_layer = self.value(current_states) 2025-12-04T09:45:29.7801620Z 2025-12-04T09:45:29.7801719Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7801953Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7802226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7802626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7802951Z res = mod(**inputs) 2025-12-04T09:45:29.7803359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7803800Z outputs = self.bert( 2025-12-04T09:45:29.7804217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7804638Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7805063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7805496Z layer_outputs = layer_module( 2025-12-04T09:45:29.7805837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7806202Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7806630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7806991Z return func(*args, **kwargs) 2025-12-04T09:45:29.7807387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7807812Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7808192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7808568Z return func(*args, **kwargs) 2025-12-04T09:45:29.7808964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.7809435Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.7809895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.7810309Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7810454Z 2025-12-04T09:45:29.7810553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7810896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7811203Z res = mod(**inputs) 2025-12-04T09:45:29.7811578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7811974Z outputs = self.bert( 2025-12-04T09:45:29.7812362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7812785Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7813185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7813592Z layer_outputs = layer_module( 2025-12-04T09:45:29.7813938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7814293Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7814672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7815043Z return func(*args, **kwargs) 2025-12-04T09:45:29.7815451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7815886Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7816308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7816766Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7817213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7817699Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7818164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.7818593Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7818727Z 2025-12-04T09:45:29.7818833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7819193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7819516Z res = mod(**inputs) 2025-12-04T09:45:29.7819914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7820346Z outputs = self.bert( 2025-12-04T09:45:29.7820749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7821179Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7821597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7822010Z layer_outputs = layer_module( 2025-12-04T09:45:29.7822361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7822785Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7823159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7823531Z return func(*args, **kwargs) 2025-12-04T09:45:29.7823951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7824392Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7824799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7825269Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7825750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7826267Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7826745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.7827250Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.7827638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.7827978Z return self.act(input) 2025-12-04T09:45:29.7828099Z 2025-12-04T09:45:29.7828205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7828571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7828902Z res = mod(**inputs) 2025-12-04T09:45:29.7829299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7829720Z outputs = self.bert( 2025-12-04T09:45:29.7830116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7830547Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7830987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7831415Z layer_outputs = layer_module( 2025-12-04T09:45:29.7831770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7832152Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7832556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7832939Z return func(*args, **kwargs) 2025-12-04T09:45:29.7833356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7833790Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7834203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7834609Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7835079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.7835589Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.7836071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.7836506Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7836646Z 2025-12-04T09:45:29.7836753Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7837122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7837452Z res = mod(**inputs) 2025-12-04T09:45:29.7837856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7838273Z outputs = self.bert( 2025-12-04T09:45:29.7838710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7839154Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7839597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7840045Z layer_outputs = layer_module( 2025-12-04T09:45:29.7840547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7840972Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7841407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7841829Z return func(*args, **kwargs) 2025-12-04T09:45:29.7842272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7842699Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7843086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7843460Z return func(*args, **kwargs) 2025-12-04T09:45:29.7843862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7844278Z self_outputs = self.self( 2025-12-04T09:45:29.7844640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7845012Z return func(*args, **kwargs) 2025-12-04T09:45:29.7845412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.7845843Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.7846007Z 2025-12-04T09:45:29.7846123Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7846484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7846828Z res = mod(**inputs) 2025-12-04T09:45:29.7847466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7847909Z outputs = self.bert( 2025-12-04T09:45:29.7848332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7848787Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7849209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7849634Z layer_outputs = layer_module( 2025-12-04T09:45:29.7850031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7850406Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7850784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7851148Z return func(*args, **kwargs) 2025-12-04T09:45:29.7851554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7851984Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7852367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7852728Z return func(*args, **kwargs) 2025-12-04T09:45:29.7853143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7853573Z self_outputs = self.self( 2025-12-04T09:45:29.7853945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7854320Z return func(*args, **kwargs) 2025-12-04T09:45:29.7854729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.7855170Z key_layer = self.key(current_states) 2025-12-04T09:45:29.7855307Z 2025-12-04T09:45:29.7855425Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7855786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7856110Z res = mod(**inputs) 2025-12-04T09:45:29.7856506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7856921Z outputs = self.bert( 2025-12-04T09:45:29.7857315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7857740Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7858162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7858573Z layer_outputs = layer_module( 2025-12-04T09:45:29.7858923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7859283Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7859654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7860023Z return func(*args, **kwargs) 2025-12-04T09:45:29.7860463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7860885Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7861280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7861642Z return func(*args, **kwargs) 2025-12-04T09:45:29.7862077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7862489Z self_outputs = self.self( 2025-12-04T09:45:29.7862852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7863219Z return func(*args, **kwargs) 2025-12-04T09:45:29.7863621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.7864076Z value_layer = self.value(current_states) 2025-12-04T09:45:29.7864217Z 2025-12-04T09:45:29.7864296Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7864507Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7864741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7865087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7865398Z res = mod(**inputs) 2025-12-04T09:45:29.7865786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7866193Z outputs = self.bert( 2025-12-04T09:45:29.7866589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7867001Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7867415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7867819Z layer_outputs = layer_module( 2025-12-04T09:45:29.7868168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7868523Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7868891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7869260Z return func(*args, **kwargs) 2025-12-04T09:45:29.7869660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7870078Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7870450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7870817Z return func(*args, **kwargs) 2025-12-04T09:45:29.7871222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.7871693Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.7872139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.7872559Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7872692Z 2025-12-04T09:45:29.7872797Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7873135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7873447Z res = mod(**inputs) 2025-12-04T09:45:29.7873838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7874272Z outputs = self.bert( 2025-12-04T09:45:29.7874663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7875124Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7875567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7875992Z layer_outputs = layer_module( 2025-12-04T09:45:29.7876342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7876711Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7877110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7877507Z return func(*args, **kwargs) 2025-12-04T09:45:29.7877952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7878446Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7878898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7879321Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7879806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7880423Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7880947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.7881420Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7881581Z 2025-12-04T09:45:29.7881700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7882115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7882416Z res = mod(**inputs) 2025-12-04T09:45:29.7882804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7883207Z outputs = self.bert( 2025-12-04T09:45:29.7883591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7883988Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7884389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7884788Z layer_outputs = layer_module( 2025-12-04T09:45:29.7885117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7885473Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7885837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7886198Z return func(*args, **kwargs) 2025-12-04T09:45:29.7886586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7887007Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7887407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7887797Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7888231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7888711Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7889172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.7889628Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.7890011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.7890356Z return self.act(input) 2025-12-04T09:45:29.7890468Z 2025-12-04T09:45:29.7890581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7890947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7891262Z res = mod(**inputs) 2025-12-04T09:45:29.7891664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7892083Z outputs = self.bert( 2025-12-04T09:45:29.7892477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7892923Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7893341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7893749Z layer_outputs = layer_module( 2025-12-04T09:45:29.7894095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7894459Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7894837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7895200Z return func(*args, **kwargs) 2025-12-04T09:45:29.7895608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7896036Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7896440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7896837Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7897294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.7897806Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.7898281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.7898720Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7898863Z 2025-12-04T09:45:29.7898969Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7899334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7899647Z res = mod(**inputs) 2025-12-04T09:45:29.7900045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7900460Z outputs = self.bert( 2025-12-04T09:45:29.7900856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7901265Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7901680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7902097Z layer_outputs = layer_module( 2025-12-04T09:45:29.7902438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7902833Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7903217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7903616Z return func(*args, **kwargs) 2025-12-04T09:45:29.7904032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7904461Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7904857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7905243Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7905673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.7906168Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.7906656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.7907067Z return input_tensor + hidden_states 2025-12-04T09:45:29.7907207Z 2025-12-04T09:45:29.7907312Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7907667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7907984Z res = mod(**inputs) 2025-12-04T09:45:29.7908379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7908808Z outputs = self.bert( 2025-12-04T09:45:29.7909199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7909613Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7910021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7910437Z layer_outputs = layer_module( 2025-12-04T09:45:29.7910786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7911141Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7911517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7911886Z return func(*args, **kwargs) 2025-12-04T09:45:29.7912294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7912715Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7913093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7913475Z return func(*args, **kwargs) 2025-12-04T09:45:29.7913882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7914290Z self_outputs = self.self( 2025-12-04T09:45:29.7914658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7915027Z return func(*args, **kwargs) 2025-12-04T09:45:29.7915425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.7915853Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.7915997Z 2025-12-04T09:45:29.7916102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7916460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7916772Z res = mod(**inputs) 2025-12-04T09:45:29.7917189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7917617Z outputs = self.bert( 2025-12-04T09:45:29.7918037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7918467Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7918893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7919330Z layer_outputs = layer_module( 2025-12-04T09:45:29.7919702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7920097Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7920627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7921075Z return func(*args, **kwargs) 2025-12-04T09:45:29.7921526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7922010Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7922416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7922791Z return func(*args, **kwargs) 2025-12-04T09:45:29.7923208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7923657Z self_outputs = self.self( 2025-12-04T09:45:29.7924048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7924438Z return func(*args, **kwargs) 2025-12-04T09:45:29.7924879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.7925342Z key_layer = self.key(current_states) 2025-12-04T09:45:29.7925483Z 2025-12-04T09:45:29.7925603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7925989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7926339Z res = mod(**inputs) 2025-12-04T09:45:29.7926767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7927207Z outputs = self.bert( 2025-12-04T09:45:29.7927634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7928089Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7928541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7928988Z layer_outputs = layer_module( 2025-12-04T09:45:29.7929364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7929755Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7930153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7930557Z return func(*args, **kwargs) 2025-12-04T09:45:29.7930993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7931456Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7931891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7934934Z return func(*args, **kwargs) 2025-12-04T09:45:29.7935387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7935842Z self_outputs = self.self( 2025-12-04T09:45:29.7936270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7936682Z return func(*args, **kwargs) 2025-12-04T09:45:29.7937127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.7937589Z value_layer = self.value(current_states) 2025-12-04T09:45:29.7937749Z 2025-12-04T09:45:29.7937839Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7938074Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.7938323Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7938746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7939134Z res = mod(**inputs) 2025-12-04T09:45:29.7939565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7940020Z outputs = self.bert( 2025-12-04T09:45:29.7940455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7940909Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7941367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7941823Z layer_outputs = layer_module( 2025-12-04T09:45:29.7942204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7942600Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7943024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7943433Z return func(*args, **kwargs) 2025-12-04T09:45:29.7943880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7944343Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7944757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7945158Z return func(*args, **kwargs) 2025-12-04T09:45:29.7945595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.7946111Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.7946626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.7947287Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7947455Z 2025-12-04T09:45:29.7947575Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7947979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7948331Z res = mod(**inputs) 2025-12-04T09:45:29.7948768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7949218Z outputs = self.bert( 2025-12-04T09:45:29.7949653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7950110Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7950624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7951166Z layer_outputs = layer_module( 2025-12-04T09:45:29.7951550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7951979Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7952391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7952799Z return func(*args, **kwargs) 2025-12-04T09:45:29.7953256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7953716Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7954162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7954623Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7955110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7955624Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7956111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.7956585Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7956738Z 2025-12-04T09:45:29.7956861Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7957259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7957619Z res = mod(**inputs) 2025-12-04T09:45:29.7958061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7958518Z outputs = self.bert( 2025-12-04T09:45:29.7958956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7959419Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7959884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7960405Z layer_outputs = layer_module( 2025-12-04T09:45:29.7960817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7961231Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7961658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7962099Z return func(*args, **kwargs) 2025-12-04T09:45:29.7962519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7962963Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7963376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7963807Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7964293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.7964814Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.7965296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.7965793Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.7966245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.7966662Z return self.act(input) 2025-12-04T09:45:29.7966789Z 2025-12-04T09:45:29.7966904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7967324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7967680Z res = mod(**inputs) 2025-12-04T09:45:29.7968105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7968556Z outputs = self.bert( 2025-12-04T09:45:29.7968990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7969447Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7969894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7970381Z layer_outputs = layer_module( 2025-12-04T09:45:29.7970769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7971167Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7971578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7971982Z return func(*args, **kwargs) 2025-12-04T09:45:29.7972429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.7972894Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.7973337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.7973768Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.7974261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.7974807Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.7975326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.7975791Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.7975942Z 2025-12-04T09:45:29.7976060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7976442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7976792Z res = mod(**inputs) 2025-12-04T09:45:29.7977224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7977672Z outputs = self.bert( 2025-12-04T09:45:29.7978107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7978565Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7979016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7979465Z layer_outputs = layer_module( 2025-12-04T09:45:29.7979843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7980238Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7980650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7981048Z return func(*args, **kwargs) 2025-12-04T09:45:29.7981527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7982031Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7982413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7982810Z return func(*args, **kwargs) 2025-12-04T09:45:29.7983228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7983655Z self_outputs = self.self( 2025-12-04T09:45:29.7984018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7984396Z return func(*args, **kwargs) 2025-12-04T09:45:29.7984809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.7985269Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.7985417Z 2025-12-04T09:45:29.7985526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7985892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7986220Z res = mod(**inputs) 2025-12-04T09:45:29.7986622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7987046Z outputs = self.bert( 2025-12-04T09:45:29.7987448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7987873Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7988293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7988724Z layer_outputs = layer_module( 2025-12-04T09:45:29.7989087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7989450Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7989859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7990264Z return func(*args, **kwargs) 2025-12-04T09:45:29.7990707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.7991165Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.7991580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7991979Z return func(*args, **kwargs) 2025-12-04T09:45:29.7992423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.7992877Z self_outputs = self.self( 2025-12-04T09:45:29.7993244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7993625Z return func(*args, **kwargs) 2025-12-04T09:45:29.7994039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.7994501Z key_layer = self.key(current_states) 2025-12-04T09:45:29.7994662Z 2025-12-04T09:45:29.7994770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.7995135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.7995473Z res = mod(**inputs) 2025-12-04T09:45:29.7995899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.7996353Z outputs = self.bert( 2025-12-04T09:45:29.7996758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.7997184Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.7997636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.7998063Z layer_outputs = layer_module( 2025-12-04T09:45:29.7998446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.7998846Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.7999262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.7999669Z return func(*args, **kwargs) 2025-12-04T09:45:29.8000131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8000803Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8001230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8001656Z return func(*args, **kwargs) 2025-12-04T09:45:29.8002088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8002544Z self_outputs = self.self( 2025-12-04T09:45:29.8002935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8003333Z return func(*args, **kwargs) 2025-12-04T09:45:29.8003766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8004233Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8004379Z 2025-12-04T09:45:29.8004475Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8004698Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8004953Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8005342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8005691Z res = mod(**inputs) 2025-12-04T09:45:29.8006114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8006569Z outputs = self.bert( 2025-12-04T09:45:29.8007000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8007470Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8007917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8008370Z layer_outputs = layer_module( 2025-12-04T09:45:29.8008747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8009132Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8009542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8009947Z return func(*args, **kwargs) 2025-12-04T09:45:29.8010391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8010851Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8011305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8011715Z return func(*args, **kwargs) 2025-12-04T09:45:29.8012179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8012701Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8013239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8013705Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8013854Z 2025-12-04T09:45:29.8013967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8014356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8014705Z res = mod(**inputs) 2025-12-04T09:45:29.8015139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8015603Z outputs = self.bert( 2025-12-04T09:45:29.8016028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8016482Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8016923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8017372Z layer_outputs = layer_module( 2025-12-04T09:45:29.8017747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8018138Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8018516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8018898Z return func(*args, **kwargs) 2025-12-04T09:45:29.8019333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8019810Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8020214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8020629Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8021119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8021636Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8022127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8022587Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8022723Z 2025-12-04T09:45:29.8022841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8023202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8023530Z res = mod(**inputs) 2025-12-04T09:45:29.8023936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8024361Z outputs = self.bert( 2025-12-04T09:45:29.8024755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8025182Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8025602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8026022Z layer_outputs = layer_module( 2025-12-04T09:45:29.8026399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8026793Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8027180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8027556Z return func(*args, **kwargs) 2025-12-04T09:45:29.8028003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8028475Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8028922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8029347Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8029836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8030408Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8030865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8031335Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8031730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8032082Z return self.act(input) 2025-12-04T09:45:29.8032197Z 2025-12-04T09:45:29.8032303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8032673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8033017Z res = mod(**inputs) 2025-12-04T09:45:29.8033449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8033896Z outputs = self.bert( 2025-12-04T09:45:29.8034318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8034757Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8035201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8035654Z layer_outputs = layer_module( 2025-12-04T09:45:29.8036026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8036415Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8036816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8037219Z return func(*args, **kwargs) 2025-12-04T09:45:29.8037661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8038119Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8038551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8038973Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8039460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8039999Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8040588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8041062Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8041212Z 2025-12-04T09:45:29.8041335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8041752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8042124Z res = mod(**inputs) 2025-12-04T09:45:29.8042555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8043019Z outputs = self.bert( 2025-12-04T09:45:29.8043439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8043887Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8044336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8044778Z layer_outputs = layer_module( 2025-12-04T09:45:29.8045156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8045607Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8046016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8046411Z return func(*args, **kwargs) 2025-12-04T09:45:29.8046841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8047460Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8047877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8048288Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8048754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8049281Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8049777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.8050222Z return input_tensor + hidden_states 2025-12-04T09:45:29.8050368Z 2025-12-04T09:45:29.8050478Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8050853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8051181Z res = mod(**inputs) 2025-12-04T09:45:29.8051593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8052022Z outputs = self.bert( 2025-12-04T09:45:29.8052426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8052865Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8053297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8053730Z layer_outputs = layer_module( 2025-12-04T09:45:29.8054087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8054467Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8054859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8055244Z return func(*args, **kwargs) 2025-12-04T09:45:29.8055662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8056108Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8056551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8056950Z return func(*args, **kwargs) 2025-12-04T09:45:29.8057371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8057798Z self_outputs = self.self( 2025-12-04T09:45:29.8058191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8058566Z return func(*args, **kwargs) 2025-12-04T09:45:29.8058986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8059432Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8059575Z 2025-12-04T09:45:29.8059689Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8060050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8060408Z res = mod(**inputs) 2025-12-04T09:45:29.8060819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8061237Z outputs = self.bert( 2025-12-04T09:45:29.8061642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8062067Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8062490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8062897Z layer_outputs = layer_module( 2025-12-04T09:45:29.8063247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8063609Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8063981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8064349Z return func(*args, **kwargs) 2025-12-04T09:45:29.8064752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8065184Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8065557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8065922Z return func(*args, **kwargs) 2025-12-04T09:45:29.8066328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8066742Z self_outputs = self.self( 2025-12-04T09:45:29.8067097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8067467Z return func(*args, **kwargs) 2025-12-04T09:45:29.8067884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8068303Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8068442Z 2025-12-04T09:45:29.8068547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8068905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8069226Z res = mod(**inputs) 2025-12-04T09:45:29.8069617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8070035Z outputs = self.bert( 2025-12-04T09:45:29.8070431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8070876Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8071299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8071716Z layer_outputs = layer_module( 2025-12-04T09:45:29.8072078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8072434Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8072855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8073235Z return func(*args, **kwargs) 2025-12-04T09:45:29.8073640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8074083Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8074463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8074854Z return func(*args, **kwargs) 2025-12-04T09:45:29.8075259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8075660Z self_outputs = self.self( 2025-12-04T09:45:29.8076010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8076366Z return func(*args, **kwargs) 2025-12-04T09:45:29.8076763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8077190Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8077331Z 2025-12-04T09:45:29.8077412Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8077626Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8077857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8078214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8078532Z res = mod(**inputs) 2025-12-04T09:45:29.8078924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8079336Z outputs = self.bert( 2025-12-04T09:45:29.8079728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8080143Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8080657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8081135Z layer_outputs = layer_module( 2025-12-04T09:45:29.8081527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8081932Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8082334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8082696Z return func(*args, **kwargs) 2025-12-04T09:45:29.8083095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8083511Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8083882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8084247Z return func(*args, **kwargs) 2025-12-04T09:45:29.8084662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8085168Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8085673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8086112Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8086262Z 2025-12-04T09:45:29.8086381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8086733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8087048Z res = mod(**inputs) 2025-12-04T09:45:29.8087448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8087855Z outputs = self.bert( 2025-12-04T09:45:29.8088252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8088697Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8089109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8089506Z layer_outputs = layer_module( 2025-12-04T09:45:29.8089842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8090197Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8090565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8090934Z return func(*args, **kwargs) 2025-12-04T09:45:29.8091336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8091764Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8092159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8092550Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8093003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8093479Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8093919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8094343Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8094477Z 2025-12-04T09:45:29.8094587Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8094934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8095251Z res = mod(**inputs) 2025-12-04T09:45:29.8095655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8096081Z outputs = self.bert( 2025-12-04T09:45:29.8096477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8096910Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8097339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8097752Z layer_outputs = layer_module( 2025-12-04T09:45:29.8098089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8098452Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8098827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8099210Z return func(*args, **kwargs) 2025-12-04T09:45:29.8099631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8100058Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8100473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8100857Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8101299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8101775Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8102222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8102671Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8103076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8103426Z return self.act(input) 2025-12-04T09:45:29.8103542Z 2025-12-04T09:45:29.8103650Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8104025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8104354Z res = mod(**inputs) 2025-12-04T09:45:29.8104763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8105177Z outputs = self.bert( 2025-12-04T09:45:29.8105587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8106020Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8106445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8106876Z layer_outputs = layer_module( 2025-12-04T09:45:29.8107233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8107606Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8107991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8108380Z return func(*args, **kwargs) 2025-12-04T09:45:29.8108852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8109294Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8109701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8110104Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8110556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8111068Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8111557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8112080Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8112219Z 2025-12-04T09:45:29.8112336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8112697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8113031Z res = mod(**inputs) 2025-12-04T09:45:29.8113463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8113913Z outputs = self.bert( 2025-12-04T09:45:29.8114310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8114737Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8115188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8115611Z layer_outputs = layer_module( 2025-12-04T09:45:29.8115966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8116340Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8116727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8117103Z return func(*args, **kwargs) 2025-12-04T09:45:29.8117549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8117993Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8118387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8118761Z return func(*args, **kwargs) 2025-12-04T09:45:29.8119177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8119606Z self_outputs = self.self( 2025-12-04T09:45:29.8119971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8120417Z return func(*args, **kwargs) 2025-12-04T09:45:29.8120856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8121338Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8121495Z 2025-12-04T09:45:29.8121618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8121992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8122324Z res = mod(**inputs) 2025-12-04T09:45:29.8122722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8123149Z outputs = self.bert( 2025-12-04T09:45:29.8123566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8123642Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8123945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8124021Z layer_outputs = layer_module( 2025-12-04T09:45:29.8124251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8124344Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8124596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8124673Z return func(*args, **kwargs) 2025-12-04T09:45:29.8124963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8125048Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8125302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8125373Z return func(*args, **kwargs) 2025-12-04T09:45:29.8125703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8125793Z self_outputs = self.self( 2025-12-04T09:45:29.8126037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8126130Z return func(*args, **kwargs) 2025-12-04T09:45:29.8126425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8126506Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8126516Z 2025-12-04T09:45:29.8126622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8126824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8126896Z res = mod(**inputs) 2025-12-04T09:45:29.8127192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8127282Z outputs = self.bert( 2025-12-04T09:45:29.8127593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8127672Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8127978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8128055Z layer_outputs = layer_module( 2025-12-04T09:45:29.8128292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8128384Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8128652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8128731Z return func(*args, **kwargs) 2025-12-04T09:45:29.8129059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8129147Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8129422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8129496Z return func(*args, **kwargs) 2025-12-04T09:45:29.8129815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8129897Z self_outputs = self.self( 2025-12-04T09:45:29.8130163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8130230Z return func(*args, **kwargs) 2025-12-04T09:45:29.8130543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8130633Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8130637Z 2025-12-04T09:45:29.8130731Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8130818Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8130934Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8131163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8131230Z res = mod(**inputs) 2025-12-04T09:45:29.8131560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8131631Z outputs = self.bert( 2025-12-04T09:45:29.8131949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8132059Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8132394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8132472Z layer_outputs = layer_module( 2025-12-04T09:45:29.8132752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8132839Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8133104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8133176Z return func(*args, **kwargs) 2025-12-04T09:45:29.8133483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8133579Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8133838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8133940Z return func(*args, **kwargs) 2025-12-04T09:45:29.8134252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8134393Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8134711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8134801Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8134805Z 2025-12-04T09:45:29.8134917Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8135143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8135213Z res = mod(**inputs) 2025-12-04T09:45:29.8135539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8135614Z outputs = self.bert( 2025-12-04T09:45:29.8135925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8136014Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8136328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8136411Z layer_outputs = layer_module( 2025-12-04T09:45:29.8136653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8136737Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8137007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8137084Z return func(*args, **kwargs) 2025-12-04T09:45:29.8137398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8137499Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8137790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8137880Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8138227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8138340Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8138660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8138750Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8138777Z 2025-12-04T09:45:29.8138919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8139137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8139318Z res = mod(**inputs) 2025-12-04T09:45:29.8139664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8139740Z outputs = self.bert( 2025-12-04T09:45:29.8140049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8140137Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8140445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8140529Z layer_outputs = layer_module( 2025-12-04T09:45:29.8140801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8140895Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8141150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8141222Z return func(*args, **kwargs) 2025-12-04T09:45:29.8141525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8141611Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8141878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8141967Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8142286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8142400Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8142700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8142818Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8143047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8143120Z return self.act(input) 2025-12-04T09:45:29.8143123Z 2025-12-04T09:45:29.8143230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8143436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8143500Z res = mod(**inputs) 2025-12-04T09:45:29.8143803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8143874Z outputs = self.bert( 2025-12-04T09:45:29.8144167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8144251Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8144541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8144614Z layer_outputs = layer_module( 2025-12-04T09:45:29.8144857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8144936Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8145187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8145259Z return func(*args, **kwargs) 2025-12-04T09:45:29.8145567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8145698Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8145964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8146067Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8146389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8146525Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8146824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8146909Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8146912Z 2025-12-04T09:45:29.8147147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8147370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8147437Z res = mod(**inputs) 2025-12-04T09:45:29.8147741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8147810Z outputs = self.bert( 2025-12-04T09:45:29.8148101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8148187Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8148477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8148557Z layer_outputs = layer_module( 2025-12-04T09:45:29.8148785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8148869Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8149124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8149195Z return func(*args, **kwargs) 2025-12-04T09:45:29.8149487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8149589Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8149862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8149947Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8150270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8150409Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8150712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.8150790Z return input_tensor + hidden_states 2025-12-04T09:45:29.8150794Z 2025-12-04T09:45:29.8150910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8151113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8151179Z res = mod(**inputs) 2025-12-04T09:45:29.8151483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8151551Z outputs = self.bert( 2025-12-04T09:45:29.8151841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8151970Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8152308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8152387Z layer_outputs = layer_module( 2025-12-04T09:45:29.8152646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8152731Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8152990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8153062Z return func(*args, **kwargs) 2025-12-04T09:45:29.8153366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8153452Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8153705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8153815Z return func(*args, **kwargs) 2025-12-04T09:45:29.8154109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8154179Z self_outputs = self.self( 2025-12-04T09:45:29.8154427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8154495Z return func(*args, **kwargs) 2025-12-04T09:45:29.8154785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8154866Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8154870Z 2025-12-04T09:45:29.8154972Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8155176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8155242Z res = mod(**inputs) 2025-12-04T09:45:29.8155534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8155600Z outputs = self.bert( 2025-12-04T09:45:29.8155885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8155964Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8156245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8156314Z layer_outputs = layer_module( 2025-12-04T09:45:29.8156542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8156621Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8156870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8156938Z return func(*args, **kwargs) 2025-12-04T09:45:29.8157221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8157313Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8157552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8157629Z return func(*args, **kwargs) 2025-12-04T09:45:29.8157914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8157984Z self_outputs = self.self( 2025-12-04T09:45:29.8158246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8158347Z return func(*args, **kwargs) 2025-12-04T09:45:29.8158630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8158720Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8158740Z 2025-12-04T09:45:29.8158847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8159052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8159116Z res = mod(**inputs) 2025-12-04T09:45:29.8159404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8159480Z outputs = self.bert( 2025-12-04T09:45:29.8159766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8159862Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8160160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8160231Z layer_outputs = layer_module( 2025-12-04T09:45:29.8160513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8160607Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8160878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8160961Z return func(*args, **kwargs) 2025-12-04T09:45:29.8161282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8161380Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8161651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8161730Z return func(*args, **kwargs) 2025-12-04T09:45:29.8162058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8162141Z self_outputs = self.self( 2025-12-04T09:45:29.8162374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8162445Z return func(*args, **kwargs) 2025-12-04T09:45:29.8162720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8162806Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8162810Z 2025-12-04T09:45:29.8162889Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8162970Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8163079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8163271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8163336Z res = mod(**inputs) 2025-12-04T09:45:29.8163634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8163700Z outputs = self.bert( 2025-12-04T09:45:29.8163992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8164066Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8164351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8164432Z layer_outputs = layer_module( 2025-12-04T09:45:29.8164679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8164785Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8165034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8165122Z return func(*args, **kwargs) 2025-12-04T09:45:29.8165440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8165528Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8165789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8165864Z return func(*args, **kwargs) 2025-12-04T09:45:29.8166156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8166314Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8166612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8166695Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8166700Z 2025-12-04T09:45:29.8166811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8167006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8167077Z res = mod(**inputs) 2025-12-04T09:45:29.8167362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8167427Z outputs = self.bert( 2025-12-04T09:45:29.8167718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8167792Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8168077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8168152Z layer_outputs = layer_module( 2025-12-04T09:45:29.8168376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8168458Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8168696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8168762Z return func(*args, **kwargs) 2025-12-04T09:45:29.8169051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8169135Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8169408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8169488Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8169816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8169929Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8170224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8170312Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8170315Z 2025-12-04T09:45:29.8170417Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8170614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8170684Z res = mod(**inputs) 2025-12-04T09:45:29.8170999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8171082Z outputs = self.bert( 2025-12-04T09:45:29.8171399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8171475Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8171778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8171848Z layer_outputs = layer_module( 2025-12-04T09:45:29.8172065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8172153Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8172391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8172481Z return func(*args, **kwargs) 2025-12-04T09:45:29.8172772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8172856Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8173120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8173198Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8173509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8173623Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8173906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8174027Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8174242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8174312Z return self.act(input) 2025-12-04T09:45:29.8174316Z 2025-12-04T09:45:29.8174427Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8174622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8174692Z res = mod(**inputs) 2025-12-04T09:45:29.8174983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8175049Z outputs = self.bert( 2025-12-04T09:45:29.8175338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8175415Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8175697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8175774Z layer_outputs = layer_module( 2025-12-04T09:45:29.8175994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8176082Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8176320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8176390Z return func(*args, **kwargs) 2025-12-04T09:45:29.8176679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8176762Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8177039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8177141Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8177453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8177605Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8177889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8177970Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8177973Z 2025-12-04T09:45:29.8178082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8178277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8178347Z res = mod(**inputs) 2025-12-04T09:45:29.8178635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8178721Z outputs = self.bert( 2025-12-04T09:45:29.8179019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8179094Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8179398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8179467Z layer_outputs = layer_module( 2025-12-04T09:45:29.8179690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8179776Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8180021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8180093Z return func(*args, **kwargs) 2025-12-04T09:45:29.8180398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8180481Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8180733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8180802Z return func(*args, **kwargs) 2025-12-04T09:45:29.8181093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8181170Z self_outputs = self.self( 2025-12-04T09:45:29.8181415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8181482Z return func(*args, **kwargs) 2025-12-04T09:45:29.8181783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8181866Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8181870Z 2025-12-04T09:45:29.8181983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8182183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8182247Z res = mod(**inputs) 2025-12-04T09:45:29.8182549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8182613Z outputs = self.bert( 2025-12-04T09:45:29.8182913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8182987Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8183305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8183398Z layer_outputs = layer_module( 2025-12-04T09:45:29.8183622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8183699Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8183964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8184034Z return func(*args, **kwargs) 2025-12-04T09:45:29.8184325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8184405Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8184641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8184734Z return func(*args, **kwargs) 2025-12-04T09:45:29.8185020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8185094Z self_outputs = self.self( 2025-12-04T09:45:29.8185335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8185403Z return func(*args, **kwargs) 2025-12-04T09:45:29.8185702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8185778Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8185782Z 2025-12-04T09:45:29.8185881Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8186077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8186139Z res = mod(**inputs) 2025-12-04T09:45:29.8186425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8186492Z outputs = self.bert( 2025-12-04T09:45:29.8186765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8186847Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8187120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8187193Z layer_outputs = layer_module( 2025-12-04T09:45:29.8187407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8187482Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8187720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8187789Z return func(*args, **kwargs) 2025-12-04T09:45:29.8188064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8188150Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8188387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8188459Z return func(*args, **kwargs) 2025-12-04T09:45:29.8188734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8188800Z self_outputs = self.self( 2025-12-04T09:45:29.8189039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8189106Z return func(*args, **kwargs) 2025-12-04T09:45:29.8189398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8189500Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8189503Z 2025-12-04T09:45:29.8189581Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8189713Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8189815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8190008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8190080Z res = mod(**inputs) 2025-12-04T09:45:29.8190360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8190432Z outputs = self.bert( 2025-12-04T09:45:29.8190711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8190801Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8191083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8191152Z layer_outputs = layer_module( 2025-12-04T09:45:29.8191366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8191450Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8191679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8191752Z return func(*args, **kwargs) 2025-12-04T09:45:29.8192027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8192105Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8192344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8192410Z return func(*args, **kwargs) 2025-12-04T09:45:29.8192696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8192820Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8193099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8193184Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8193187Z 2025-12-04T09:45:29.8193287Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8193477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8193545Z res = mod(**inputs) 2025-12-04T09:45:29.8193822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8193896Z outputs = self.bert( 2025-12-04T09:45:29.8194172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8194242Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8194529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8194596Z layer_outputs = layer_module( 2025-12-04T09:45:29.8194816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8194892Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8195149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8195238Z return func(*args, **kwargs) 2025-12-04T09:45:29.8195521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8195601Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8195884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8195960Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8196269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8196371Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8196647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8196757Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8196762Z 2025-12-04T09:45:29.8196862Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8197058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8197120Z res = mod(**inputs) 2025-12-04T09:45:29.8197402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8197473Z outputs = self.bert( 2025-12-04T09:45:29.8197753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8197826Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8198118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8198191Z layer_outputs = layer_module( 2025-12-04T09:45:29.8198426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8198505Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8198756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8198838Z return func(*args, **kwargs) 2025-12-04T09:45:29.8199147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8199244Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8199525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8199608Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8199958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8200071Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8200450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8200591Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8200824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8200907Z return self.act(input) 2025-12-04T09:45:29.8200911Z 2025-12-04T09:45:29.8201023Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8201243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8201331Z res = mod(**inputs) 2025-12-04T09:45:29.8201649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8201745Z outputs = self.bert( 2025-12-04T09:45:29.8202058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8202138Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8202480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8202560Z layer_outputs = layer_module( 2025-12-04T09:45:29.8202803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8202899Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8203165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8203278Z return func(*args, **kwargs) 2025-12-04T09:45:29.8203590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8203680Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8203969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8204050Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8204397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8204539Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8204849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8204949Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8204954Z 2025-12-04T09:45:29.8205067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8205281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8205356Z res = mod(**inputs) 2025-12-04T09:45:29.8205672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8205751Z outputs = self.bert( 2025-12-04T09:45:29.8206061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8206140Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8206459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8206535Z layer_outputs = layer_module( 2025-12-04T09:45:29.8206786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8206874Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8207138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8207222Z return func(*args, **kwargs) 2025-12-04T09:45:29.8207530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8207621Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8207910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8207994Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8208360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8208520Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8208833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.8208942Z return input_tensor + hidden_states 2025-12-04T09:45:29.8208947Z 2025-12-04T09:45:29.8209062Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8209291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8209361Z res = mod(**inputs) 2025-12-04T09:45:29.8209671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8209750Z outputs = self.bert( 2025-12-04T09:45:29.8210118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8210216Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8210604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8210688Z layer_outputs = layer_module( 2025-12-04T09:45:29.8210920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8210999Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8211245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8211320Z return func(*args, **kwargs) 2025-12-04T09:45:29.8211612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8211703Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8211945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8212016Z return func(*args, **kwargs) 2025-12-04T09:45:29.8212313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8212383Z self_outputs = self.self( 2025-12-04T09:45:29.8212629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8212704Z return func(*args, **kwargs) 2025-12-04T09:45:29.8213016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8213105Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8213108Z 2025-12-04T09:45:29.8213213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8213411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8213486Z res = mod(**inputs) 2025-12-04T09:45:29.8213776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8213850Z outputs = self.bert( 2025-12-04T09:45:29.8214142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8214216Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8214521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8214592Z layer_outputs = layer_module( 2025-12-04T09:45:29.8214845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8214934Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8215188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8215264Z return func(*args, **kwargs) 2025-12-04T09:45:29.8215565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8215652Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8215903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8215973Z return func(*args, **kwargs) 2025-12-04T09:45:29.8216267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8216338Z self_outputs = self.self( 2025-12-04T09:45:29.8216598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8216674Z return func(*args, **kwargs) 2025-12-04T09:45:29.8216958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8217038Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8217041Z 2025-12-04T09:45:29.8217149Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8217347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8217417Z res = mod(**inputs) 2025-12-04T09:45:29.8217705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8217772Z outputs = self.bert( 2025-12-04T09:45:29.8218066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8218141Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8218427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8218503Z layer_outputs = layer_module( 2025-12-04T09:45:29.8218722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8218806Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8219046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8219116Z return func(*args, **kwargs) 2025-12-04T09:45:29.8219408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8219491Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8219738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8219806Z return func(*args, **kwargs) 2025-12-04T09:45:29.8220092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8220167Z self_outputs = self.self( 2025-12-04T09:45:29.8220408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8220475Z return func(*args, **kwargs) 2025-12-04T09:45:29.8220768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8220848Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8220852Z 2025-12-04T09:45:29.8220954Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8221052Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8221156Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8221362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8221428Z res = mod(**inputs) 2025-12-04T09:45:29.8221728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8221805Z outputs = self.bert( 2025-12-04T09:45:29.8222087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8222168Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8222452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8222543Z layer_outputs = layer_module( 2025-12-04T09:45:29.8222773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8222850Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8223095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8223163Z return func(*args, **kwargs) 2025-12-04T09:45:29.8223447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8223537Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8223773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8223840Z return func(*args, **kwargs) 2025-12-04T09:45:29.8224150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8224290Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8224605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8224695Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8224698Z 2025-12-04T09:45:29.8224808Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8225027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8225096Z res = mod(**inputs) 2025-12-04T09:45:29.8225415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8225486Z outputs = self.bert( 2025-12-04T09:45:29.8225795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8225884Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8226184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8226257Z layer_outputs = layer_module( 2025-12-04T09:45:29.8226489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8226567Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8226819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8226886Z return func(*args, **kwargs) 2025-12-04T09:45:29.8227179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8227296Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8227578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8227664Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8228006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8228114Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8228432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8228514Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8228517Z 2025-12-04T09:45:29.8228627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8228825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8228907Z res = mod(**inputs) 2025-12-04T09:45:29.8229221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8229289Z outputs = self.bert( 2025-12-04T09:45:29.8229595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8229678Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8229977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8230054Z layer_outputs = layer_module( 2025-12-04T09:45:29.8230286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8230367Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8230626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8230698Z return func(*args, **kwargs) 2025-12-04T09:45:29.8230995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8231087Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8231362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8231449Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8231790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8231894Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8232204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8232321Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8232553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8232623Z return self.act(input) 2025-12-04T09:45:29.8232626Z 2025-12-04T09:45:29.8232732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8232943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8233007Z res = mod(**inputs) 2025-12-04T09:45:29.8233311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8233386Z outputs = self.bert( 2025-12-04T09:45:29.8233704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8233804Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8234094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8234165Z layer_outputs = layer_module( 2025-12-04T09:45:29.8234415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8234498Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8234748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8234818Z return func(*args, **kwargs) 2025-12-04T09:45:29.8235104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8235196Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8235479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8235558Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8235891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8236025Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8236324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8236409Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8236412Z 2025-12-04T09:45:29.8236519Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8236729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8236796Z res = mod(**inputs) 2025-12-04T09:45:29.8237102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8237172Z outputs = self.bert( 2025-12-04T09:45:29.8237463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8237550Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8237857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8237939Z layer_outputs = layer_module( 2025-12-04T09:45:29.8238181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8238267Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8238537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8238614Z return func(*args, **kwargs) 2025-12-04T09:45:29.8238925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8239023Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8239285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8239366Z return func(*args, **kwargs) 2025-12-04T09:45:29.8239676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8239752Z self_outputs = self.self( 2025-12-04T09:45:29.8240020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8240093Z return func(*args, **kwargs) 2025-12-04T09:45:29.8240496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8240621Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8240625Z 2025-12-04T09:45:29.8240740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8240980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8241052Z res = mod(**inputs) 2025-12-04T09:45:29.8241363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8241445Z outputs = self.bert( 2025-12-04T09:45:29.8241752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8241842Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8242175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8242249Z layer_outputs = layer_module( 2025-12-04T09:45:29.8242481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8242564Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8242815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8242895Z return func(*args, **kwargs) 2025-12-04T09:45:29.8243186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8243278Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8243524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8243598Z return func(*args, **kwargs) 2025-12-04T09:45:29.8243897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8243969Z self_outputs = self.self( 2025-12-04T09:45:29.8244221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8244290Z return func(*args, **kwargs) 2025-12-04T09:45:29.8244578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8244662Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8244666Z 2025-12-04T09:45:29.8244770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8244970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8245046Z res = mod(**inputs) 2025-12-04T09:45:29.8245340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8245412Z outputs = self.bert( 2025-12-04T09:45:29.8245702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8245776Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8246072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8246144Z layer_outputs = layer_module( 2025-12-04T09:45:29.8246378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8246457Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8246733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8246833Z return func(*args, **kwargs) 2025-12-04T09:45:29.8247377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8247513Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8247774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8247844Z return func(*args, **kwargs) 2025-12-04T09:45:29.8248145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8248215Z self_outputs = self.self( 2025-12-04T09:45:29.8248457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8248571Z return func(*args, **kwargs) 2025-12-04T09:45:29.8248874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8248955Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8248968Z 2025-12-04T09:45:29.8249050Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8249133Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8249248Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8249456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8249522Z res = mod(**inputs) 2025-12-04T09:45:29.8249836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8249904Z outputs = self.bert( 2025-12-04T09:45:29.8250217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8250293Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8250593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8250673Z layer_outputs = layer_module( 2025-12-04T09:45:29.8250910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8250989Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8251248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8251319Z return func(*args, **kwargs) 2025-12-04T09:45:29.8251635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8251720Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8251970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8252044Z return func(*args, **kwargs) 2025-12-04T09:45:29.8252341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8252479Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8252771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8252854Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8252857Z 2025-12-04T09:45:29.8252966Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8253167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8253264Z res = mod(**inputs) 2025-12-04T09:45:29.8253603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8253669Z outputs = self.bert( 2025-12-04T09:45:29.8253985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8254061Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8254353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8254433Z layer_outputs = layer_module( 2025-12-04T09:45:29.8254658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8254746Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8254996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8255090Z return func(*args, **kwargs) 2025-12-04T09:45:29.8255404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8255492Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8255769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8255855Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8256191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8256315Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8256614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8256698Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8256702Z 2025-12-04T09:45:29.8256813Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8257015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8257090Z res = mod(**inputs) 2025-12-04T09:45:29.8257393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8257460Z outputs = self.bert( 2025-12-04T09:45:29.8257775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8257850Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8258157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8258239Z layer_outputs = layer_module( 2025-12-04T09:45:29.8258471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8258558Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8258812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8258883Z return func(*args, **kwargs) 2025-12-04T09:45:29.8259191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8259274Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8259558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8259635Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8259987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8260117Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8260428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8260545Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8260769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8260841Z return self.act(input) 2025-12-04T09:45:29.8260845Z 2025-12-04T09:45:29.8260957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8261161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8261226Z res = mod(**inputs) 2025-12-04T09:45:29.8261556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8261626Z outputs = self.bert( 2025-12-04T09:45:29.8261929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8262006Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8262299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8262377Z layer_outputs = layer_module( 2025-12-04T09:45:29.8262605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8262684Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8262940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8263013Z return func(*args, **kwargs) 2025-12-04T09:45:29.8263312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8263396Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8263662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8263748Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8264078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8264219Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8264514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8264602Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8264607Z 2025-12-04T09:45:29.8264719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8264922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8264986Z res = mod(**inputs) 2025-12-04T09:45:29.8265290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8265358Z outputs = self.bert( 2025-12-04T09:45:29.8265668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8265745Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8266058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8266162Z layer_outputs = layer_module( 2025-12-04T09:45:29.8266417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8266506Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8266780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8266856Z return func(*args, **kwargs) 2025-12-04T09:45:29.8267171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8267261Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8267540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8267629Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8267967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8268140Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8268451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.8268541Z return input_tensor + hidden_states 2025-12-04T09:45:29.8268545Z 2025-12-04T09:45:29.8268665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8268877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8268955Z res = mod(**inputs) 2025-12-04T09:45:29.8269269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8269342Z outputs = self.bert( 2025-12-04T09:45:29.8269662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8269748Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8270057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8270144Z layer_outputs = layer_module( 2025-12-04T09:45:29.8270385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8270485Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8270732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8270805Z return func(*args, **kwargs) 2025-12-04T09:45:29.8271109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8271204Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8271476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8271552Z return func(*args, **kwargs) 2025-12-04T09:45:29.8271869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8271951Z self_outputs = self.self( 2025-12-04T09:45:29.8272195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8272268Z return func(*args, **kwargs) 2025-12-04T09:45:29.8272570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8272654Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8272659Z 2025-12-04T09:45:29.8272790Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8273010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8273077Z res = mod(**inputs) 2025-12-04T09:45:29.8273396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8273466Z outputs = self.bert( 2025-12-04T09:45:29.8273783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8273863Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8274172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8274254Z layer_outputs = layer_module( 2025-12-04T09:45:29.8274496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8274603Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8274882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8274955Z return func(*args, **kwargs) 2025-12-04T09:45:29.8275273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8275369Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8275612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8275688Z return func(*args, **kwargs) 2025-12-04T09:45:29.8275980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8276053Z self_outputs = self.self( 2025-12-04T09:45:29.8276304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8276373Z return func(*args, **kwargs) 2025-12-04T09:45:29.8276682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8276768Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8276772Z 2025-12-04T09:45:29.8276884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8277104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8277173Z res = mod(**inputs) 2025-12-04T09:45:29.8277492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8277564Z outputs = self.bert( 2025-12-04T09:45:29.8277876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8277964Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8278272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8278348Z layer_outputs = layer_module( 2025-12-04T09:45:29.8278596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8278678Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8278942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8279016Z return func(*args, **kwargs) 2025-12-04T09:45:29.8279359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8279473Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8279734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8279815Z return func(*args, **kwargs) 2025-12-04T09:45:29.8280137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8280213Z self_outputs = self.self( 2025-12-04T09:45:29.8280558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8280637Z return func(*args, **kwargs) 2025-12-04T09:45:29.8280950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8281045Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8281074Z 2025-12-04T09:45:29.8281165Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8281260Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8281373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8281592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8281670Z res = mod(**inputs) 2025-12-04T09:45:29.8281985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8282056Z outputs = self.bert( 2025-12-04T09:45:29.8282375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8282455Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8282775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8282855Z layer_outputs = layer_module( 2025-12-04T09:45:29.8283097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8283190Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8283455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8283534Z return func(*args, **kwargs) 2025-12-04T09:45:29.8283844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8283933Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8284199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8284272Z return func(*args, **kwargs) 2025-12-04T09:45:29.8284585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8284734Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8285045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8285142Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8285146Z 2025-12-04T09:45:29.8285256Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8285473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8285548Z res = mod(**inputs) 2025-12-04T09:45:29.8285860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8285938Z outputs = self.bert( 2025-12-04T09:45:29.8286270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8286368Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8286709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8286787Z layer_outputs = layer_module( 2025-12-04T09:45:29.8287025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8287118Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8287375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8287455Z return func(*args, **kwargs) 2025-12-04T09:45:29.8287765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8287878Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8288170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8288260Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8288586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8288691Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8288976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8289065Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8289068Z 2025-12-04T09:45:29.8289168Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8289370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8289437Z res = mod(**inputs) 2025-12-04T09:45:29.8289725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8289799Z outputs = self.bert( 2025-12-04T09:45:29.8290087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8290162Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8290461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8290533Z layer_outputs = layer_module( 2025-12-04T09:45:29.8290765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8290846Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8291096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8291174Z return func(*args, **kwargs) 2025-12-04T09:45:29.8291479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8291560Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8291828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8291905Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8292236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8292344Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8292664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8292831Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8293046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8293137Z return self.act(input) 2025-12-04T09:45:29.8293141Z 2025-12-04T09:45:29.8293245Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8293443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8293516Z res = mod(**inputs) 2025-12-04T09:45:29.8293803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8293870Z outputs = self.bert( 2025-12-04T09:45:29.8294169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8294261Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8294559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8294631Z layer_outputs = layer_module( 2025-12-04T09:45:29.8294856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8294943Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8295189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8295264Z return func(*args, **kwargs) 2025-12-04T09:45:29.8295552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8295639Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8295911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8295988Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8296309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8296449Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8296741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8296830Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8296833Z 2025-12-04T09:45:29.8296935Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8297136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8297210Z res = mod(**inputs) 2025-12-04T09:45:29.8297502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8297573Z outputs = self.bert( 2025-12-04T09:45:29.8297864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8297937Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8298234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8298303Z layer_outputs = layer_module( 2025-12-04T09:45:29.8298525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8298611Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8298871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8298962Z return func(*args, **kwargs) 2025-12-04T09:45:29.8299254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8299350Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8299592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8299660Z return func(*args, **kwargs) 2025-12-04T09:45:29.8299946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8300015Z self_outputs = self.self( 2025-12-04T09:45:29.8300251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8300357Z return func(*args, **kwargs) 2025-12-04T09:45:29.8300643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8300723Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8300732Z 2025-12-04T09:45:29.8300836Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8301030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8301101Z res = mod(**inputs) 2025-12-04T09:45:29.8301383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8301448Z outputs = self.bert( 2025-12-04T09:45:29.8301738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8301812Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8302101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8302171Z layer_outputs = layer_module( 2025-12-04T09:45:29.8302391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8302478Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8302714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8302781Z return func(*args, **kwargs) 2025-12-04T09:45:29.8303071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8303153Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8303398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8303469Z return func(*args, **kwargs) 2025-12-04T09:45:29.8303749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8303825Z self_outputs = self.self( 2025-12-04T09:45:29.8304060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8304135Z return func(*args, **kwargs) 2025-12-04T09:45:29.8304415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8304493Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8304496Z 2025-12-04T09:45:29.8304606Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8304819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8304901Z res = mod(**inputs) 2025-12-04T09:45:29.8305202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8305268Z outputs = self.bert( 2025-12-04T09:45:29.8305579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8305655Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8305937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8306015Z layer_outputs = layer_module( 2025-12-04T09:45:29.8306235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8306319Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8306574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8306643Z return func(*args, **kwargs) 2025-12-04T09:45:29.8306932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8307012Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8307250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8307325Z return func(*args, **kwargs) 2025-12-04T09:45:29.8307605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8307679Z self_outputs = self.self( 2025-12-04T09:45:29.8307915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8307986Z return func(*args, **kwargs) 2025-12-04T09:45:29.8308275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8308354Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8308357Z 2025-12-04T09:45:29.8308443Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8308520Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8308622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8308826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8308888Z res = mod(**inputs) 2025-12-04T09:45:29.8309174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8309248Z outputs = self.bert( 2025-12-04T09:45:29.8309531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8309613Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8309895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8309967Z layer_outputs = layer_module( 2025-12-04T09:45:29.8310192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8310268Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8310504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8310579Z return func(*args, **kwargs) 2025-12-04T09:45:29.8310875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8310982Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8311233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8311299Z return func(*args, **kwargs) 2025-12-04T09:45:29.8311598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8311723Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8312012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8312093Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8312096Z 2025-12-04T09:45:29.8312195Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8312397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8312480Z res = mod(**inputs) 2025-12-04T09:45:29.8312767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8312839Z outputs = self.bert( 2025-12-04T09:45:29.8313122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8313199Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8313480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8313550Z layer_outputs = layer_module( 2025-12-04T09:45:29.8313779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8313856Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8314095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8314169Z return func(*args, **kwargs) 2025-12-04T09:45:29.8314454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8314541Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8314800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8314874Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8315194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8315295Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8315586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8315670Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8315673Z 2025-12-04T09:45:29.8315775Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8315981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8316046Z res = mod(**inputs) 2025-12-04T09:45:29.8316349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8316415Z outputs = self.bert( 2025-12-04T09:45:29.8316705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8316783Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8317091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8317179Z layer_outputs = layer_module( 2025-12-04T09:45:29.8317411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8317490Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8317757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8317827Z return func(*args, **kwargs) 2025-12-04T09:45:29.8318112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8318204Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8318462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8318563Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8318874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8318975Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8319263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8319374Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8319590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8319668Z return self.act(input) 2025-12-04T09:45:29.8319672Z 2025-12-04T09:45:29.8319777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8319985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8320058Z res = mod(**inputs) 2025-12-04T09:45:29.8320448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8320542Z outputs = self.bert( 2025-12-04T09:45:29.8320862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8320950Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8321269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8321347Z layer_outputs = layer_module( 2025-12-04T09:45:29.8321609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8321687Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8321921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8322001Z return func(*args, **kwargs) 2025-12-04T09:45:29.8322275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8322368Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8322618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8322694Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8323007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8323134Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8323438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8323537Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8323541Z 2025-12-04T09:45:29.8323641Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8323839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8323930Z res = mod(**inputs) 2025-12-04T09:45:29.8324218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8324293Z outputs = self.bert( 2025-12-04T09:45:29.8324574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8324653Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8324934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8325029Z layer_outputs = layer_module( 2025-12-04T09:45:29.8325271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8325348Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8325594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8325661Z return func(*args, **kwargs) 2025-12-04T09:45:29.8325950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8326038Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8326302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8326376Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8326705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8326837Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8327146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.8327221Z return input_tensor + hidden_states 2025-12-04T09:45:29.8327224Z 2025-12-04T09:45:29.8327325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8327528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8327594Z res = mod(**inputs) 2025-12-04T09:45:29.8327898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8327965Z outputs = self.bert( 2025-12-04T09:45:29.8328255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8328335Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8328630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8328699Z layer_outputs = layer_module( 2025-12-04T09:45:29.8328931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8329010Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8329263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8329332Z return func(*args, **kwargs) 2025-12-04T09:45:29.8329642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8329749Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8329983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8330055Z return func(*args, **kwargs) 2025-12-04T09:45:29.8330352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8330423Z self_outputs = self.self( 2025-12-04T09:45:29.8330666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8330733Z return func(*args, **kwargs) 2025-12-04T09:45:29.8331013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8331102Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8331122Z 2025-12-04T09:45:29.8331227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8331429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8331492Z res = mod(**inputs) 2025-12-04T09:45:29.8331778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8331853Z outputs = self.bert( 2025-12-04T09:45:29.8332135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8332215Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8332492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8332561Z layer_outputs = layer_module( 2025-12-04T09:45:29.8332792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8332872Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8333110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8333188Z return func(*args, **kwargs) 2025-12-04T09:45:29.8333466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8333555Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8333791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8333860Z return func(*args, **kwargs) 2025-12-04T09:45:29.8334150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8334221Z self_outputs = self.self( 2025-12-04T09:45:29.8334459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8334531Z return func(*args, **kwargs) 2025-12-04T09:45:29.8334816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8334903Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8334906Z 2025-12-04T09:45:29.8335009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8335209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8335280Z res = mod(**inputs) 2025-12-04T09:45:29.8335568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8335660Z outputs = self.bert( 2025-12-04T09:45:29.8335959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8336032Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8336339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8336411Z layer_outputs = layer_module( 2025-12-04T09:45:29.8336628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8336714Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8336950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8337023Z return func(*args, **kwargs) 2025-12-04T09:45:29.8337303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8337404Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8337650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8337718Z return func(*args, **kwargs) 2025-12-04T09:45:29.8338007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8338075Z self_outputs = self.self( 2025-12-04T09:45:29.8338310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8338386Z return func(*args, **kwargs) 2025-12-04T09:45:29.8338666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8338753Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8338766Z 2025-12-04T09:45:29.8338845Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8338923Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8339030Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8339227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8339291Z res = mod(**inputs) 2025-12-04T09:45:29.8339583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8339649Z outputs = self.bert( 2025-12-04T09:45:29.8339932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8340012Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8340302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8340379Z layer_outputs = layer_module( 2025-12-04T09:45:29.8340590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8340665Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8340906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8340972Z return func(*args, **kwargs) 2025-12-04T09:45:29.8341255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8341335Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8341565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8341656Z return func(*args, **kwargs) 2025-12-04T09:45:29.8341946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8342069Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8342366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8342451Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8342455Z 2025-12-04T09:45:29.8342561Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8342752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8342815Z res = mod(**inputs) 2025-12-04T09:45:29.8343101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8343184Z outputs = self.bert( 2025-12-04T09:45:29.8343470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8343540Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8343813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8343890Z layer_outputs = layer_module( 2025-12-04T09:45:29.8344101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8344176Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8344414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8344479Z return func(*args, **kwargs) 2025-12-04T09:45:29.8344759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8344842Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8345092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8345176Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8345479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8345585Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8345859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8345938Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8345941Z 2025-12-04T09:45:29.8346048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8346241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8346304Z res = mod(**inputs) 2025-12-04T09:45:29.8346585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8346650Z outputs = self.bert( 2025-12-04T09:45:29.8346927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8346999Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8347395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8347478Z layer_outputs = layer_module( 2025-12-04T09:45:29.8347734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8347823Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8348087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8348155Z return func(*args, **kwargs) 2025-12-04T09:45:29.8348479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8348564Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8348821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8348905Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8349216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8349329Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8349638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8349760Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8349977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8350044Z return self.act(input) 2025-12-04T09:45:29.8350048Z 2025-12-04T09:45:29.8350154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8350347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8350409Z res = mod(**inputs) 2025-12-04T09:45:29.8350693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8350758Z outputs = self.bert( 2025-12-04T09:45:29.8351039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8351120Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8351396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8351473Z layer_outputs = layer_module( 2025-12-04T09:45:29.8351686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8351762Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8352002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8352068Z return func(*args, **kwargs) 2025-12-04T09:45:29.8352356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8352441Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8352692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8352773Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8353133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8353261Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8353549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8353628Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8353631Z 2025-12-04T09:45:29.8353737Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8353949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8354033Z res = mod(**inputs) 2025-12-04T09:45:29.8354321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8354387Z outputs = self.bert( 2025-12-04T09:45:29.8354688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8354760Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8355036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8355110Z layer_outputs = layer_module( 2025-12-04T09:45:29.8355325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8355403Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8355661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8355729Z return func(*args, **kwargs) 2025-12-04T09:45:29.8356014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8356093Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8356325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8356399Z return func(*args, **kwargs) 2025-12-04T09:45:29.8356676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8356753Z self_outputs = self.self( 2025-12-04T09:45:29.8356993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8357064Z return func(*args, **kwargs) 2025-12-04T09:45:29.8357354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8357434Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8357439Z 2025-12-04T09:45:29.8357543Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8357747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8357811Z res = mod(**inputs) 2025-12-04T09:45:29.8358103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8358168Z outputs = self.bert( 2025-12-04T09:45:29.8358453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8358535Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8358818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8358893Z layer_outputs = layer_module( 2025-12-04T09:45:29.8359114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8359196Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8359465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8359538Z return func(*args, **kwargs) 2025-12-04T09:45:29.8359854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8359944Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8360207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8360299Z return func(*args, **kwargs) 2025-12-04T09:45:29.8360649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8360741Z self_outputs = self.self( 2025-12-04T09:45:29.8361007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8361081Z return func(*args, **kwargs) 2025-12-04T09:45:29.8361397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8361481Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8361485Z 2025-12-04T09:45:29.8361607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8361841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8361908Z res = mod(**inputs) 2025-12-04T09:45:29.8362199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8362277Z outputs = self.bert( 2025-12-04T09:45:29.8362566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8362647Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8362932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8363004Z layer_outputs = layer_module( 2025-12-04T09:45:29.8363242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8363322Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8363561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8363640Z return func(*args, **kwargs) 2025-12-04T09:45:29.8363922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8364011Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8364248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8364313Z return func(*args, **kwargs) 2025-12-04T09:45:29.8364602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8364670Z self_outputs = self.self( 2025-12-04T09:45:29.8364917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8364986Z return func(*args, **kwargs) 2025-12-04T09:45:29.8365266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8365354Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8365357Z 2025-12-04T09:45:29.8365437Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8365516Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8365624Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8365819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8365890Z res = mod(**inputs) 2025-12-04T09:45:29.8366176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8366260Z outputs = self.bert( 2025-12-04T09:45:29.8366574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8366648Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8366956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8367028Z layer_outputs = layer_module( 2025-12-04T09:45:29.8367246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8367331Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8367567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8367636Z return func(*args, **kwargs) 2025-12-04T09:45:29.8367926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8368029Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8368273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8368343Z return func(*args, **kwargs) 2025-12-04T09:45:29.8368622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8368753Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8369036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8369124Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8369127Z 2025-12-04T09:45:29.8369230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8369425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8369499Z res = mod(**inputs) 2025-12-04T09:45:29.8369786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8369854Z outputs = self.bert( 2025-12-04T09:45:29.8370140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8370211Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8370497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8370566Z layer_outputs = layer_module( 2025-12-04T09:45:29.8370784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8370873Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8371111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8371178Z return func(*args, **kwargs) 2025-12-04T09:45:29.8371466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8371549Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8371811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8371887Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8372196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8372309Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8372619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8372728Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8372732Z 2025-12-04T09:45:29.8372837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8373051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8373127Z res = mod(**inputs) 2025-12-04T09:45:29.8373424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8373499Z outputs = self.bert( 2025-12-04T09:45:29.8373787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8373862Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8374171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8374242Z layer_outputs = layer_module( 2025-12-04T09:45:29.8374460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8374548Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8374788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8374861Z return func(*args, **kwargs) 2025-12-04T09:45:29.8375142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8375225Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8375501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8375579Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8375891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8375990Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8376273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8376391Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8376603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8376672Z return self.act(input) 2025-12-04T09:45:29.8376675Z 2025-12-04T09:45:29.8376783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8376979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8377050Z res = mod(**inputs) 2025-12-04T09:45:29.8377341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8377409Z outputs = self.bert( 2025-12-04T09:45:29.8377703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8377773Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8378062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8378131Z layer_outputs = layer_module( 2025-12-04T09:45:29.8378351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8378439Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8378711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8378797Z return func(*args, **kwargs) 2025-12-04T09:45:29.8379091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8379188Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8379454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8379530Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8379848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8379994Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8380292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8380402Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8380409Z 2025-12-04T09:45:29.8380525Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8380725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8380796Z res = mod(**inputs) 2025-12-04T09:45:29.8381089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8381154Z outputs = self.bert( 2025-12-04T09:45:29.8381452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8381524Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8381826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8381898Z layer_outputs = layer_module( 2025-12-04T09:45:29.8382117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8382202Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8382440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8382514Z return func(*args, **kwargs) 2025-12-04T09:45:29.8382798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8382878Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8383148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8383225Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8383544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8383683Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8383977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.8384061Z return input_tensor + hidden_states 2025-12-04T09:45:29.8384064Z 2025-12-04T09:45:29.8384168Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8384365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8384437Z res = mod(**inputs) 2025-12-04T09:45:29.8384757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8384832Z outputs = self.bert( 2025-12-04T09:45:29.8385119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8385191Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8385485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8385555Z layer_outputs = layer_module( 2025-12-04T09:45:29.8385767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8385850Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8386081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8386155Z return func(*args, **kwargs) 2025-12-04T09:45:29.8386430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8386529Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8386774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8386842Z return func(*args, **kwargs) 2025-12-04T09:45:29.8387132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8387211Z self_outputs = self.self( 2025-12-04T09:45:29.8387441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8387515Z return func(*args, **kwargs) 2025-12-04T09:45:29.8387791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8387872Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8387877Z 2025-12-04T09:45:29.8387986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8388176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8388246Z res = mod(**inputs) 2025-12-04T09:45:29.8388531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8388597Z outputs = self.bert( 2025-12-04T09:45:29.8388887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8388968Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8389251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8389321Z layer_outputs = layer_module( 2025-12-04T09:45:29.8389535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8389618Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8389850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8389915Z return func(*args, **kwargs) 2025-12-04T09:45:29.8390199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8390279Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8390520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8390587Z return func(*args, **kwargs) 2025-12-04T09:45:29.8390923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8391020Z self_outputs = self.self( 2025-12-04T09:45:29.8391257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8391324Z return func(*args, **kwargs) 2025-12-04T09:45:29.8391630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8391709Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8391712Z 2025-12-04T09:45:29.8391821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8392025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8392087Z res = mod(**inputs) 2025-12-04T09:45:29.8392383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8392487Z outputs = self.bert( 2025-12-04T09:45:29.8392776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8392848Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8393131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8393209Z layer_outputs = layer_module( 2025-12-04T09:45:29.8393428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8393507Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8393753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8393820Z return func(*args, **kwargs) 2025-12-04T09:45:29.8394111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8394193Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8394432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8394509Z return func(*args, **kwargs) 2025-12-04T09:45:29.8394788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8394863Z self_outputs = self.self( 2025-12-04T09:45:29.8395099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8395164Z return func(*args, **kwargs) 2025-12-04T09:45:29.8395456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8395537Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8395540Z 2025-12-04T09:45:29.8395621Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8395709Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8395813Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8396020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8396086Z res = mod(**inputs) 2025-12-04T09:45:29.8396391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8396471Z outputs = self.bert( 2025-12-04T09:45:29.8396776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8396857Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8397189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8397282Z layer_outputs = layer_module( 2025-12-04T09:45:29.8397528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8397629Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8397889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8397971Z return func(*args, **kwargs) 2025-12-04T09:45:29.8398280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8398374Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8398635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8398730Z return func(*args, **kwargs) 2025-12-04T09:45:29.8399050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8399188Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8399502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8399601Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8399605Z 2025-12-04T09:45:29.8399715Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8399938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8400008Z res = mod(**inputs) 2025-12-04T09:45:29.8400389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8400486Z outputs = self.bert( 2025-12-04T09:45:29.8400815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8400906Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8401235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8401314Z layer_outputs = layer_module( 2025-12-04T09:45:29.8401580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8401669Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8401946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8402045Z return func(*args, **kwargs) 2025-12-04T09:45:29.8402334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8402428Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8402687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8402764Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8403087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8403191Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8403483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8403566Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8403571Z 2025-12-04T09:45:29.8403696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8403920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8403986Z res = mod(**inputs) 2025-12-04T09:45:29.8404288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8404362Z outputs = self.bert( 2025-12-04T09:45:29.8404647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8404726Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8405011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8405081Z layer_outputs = layer_module( 2025-12-04T09:45:29.8405311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8405412Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8405669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8405736Z return func(*args, **kwargs) 2025-12-04T09:45:29.8406024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8406114Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8406374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8406448Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8406767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8406877Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8407177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8407291Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8407522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8407600Z return self.act(input) 2025-12-04T09:45:29.8407604Z 2025-12-04T09:45:29.8407705Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8407909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8407974Z res = mod(**inputs) 2025-12-04T09:45:29.8408268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8408341Z outputs = self.bert( 2025-12-04T09:45:29.8408626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8408697Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8408986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8409055Z layer_outputs = layer_module( 2025-12-04T09:45:29.8409282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8409363Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8409609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8409686Z return func(*args, **kwargs) 2025-12-04T09:45:29.8409994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8410103Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8410368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8410458Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8410786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8410920Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8411214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8411304Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8411307Z 2025-12-04T09:45:29.8411414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8411642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8411711Z res = mod(**inputs) 2025-12-04T09:45:29.8412005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8412085Z outputs = self.bert( 2025-12-04T09:45:29.8412376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8412456Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8412749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8412821Z layer_outputs = layer_module( 2025-12-04T09:45:29.8413057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8413142Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8413388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8413464Z return func(*args, **kwargs) 2025-12-04T09:45:29.8413759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8413850Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8414096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8414164Z return func(*args, **kwargs) 2025-12-04T09:45:29.8414462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8414533Z self_outputs = self.self( 2025-12-04T09:45:29.8414788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8414858Z return func(*args, **kwargs) 2025-12-04T09:45:29.8415148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8415240Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8415244Z 2025-12-04T09:45:29.8415350Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8415550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8415624Z res = mod(**inputs) 2025-12-04T09:45:29.8415921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8415996Z outputs = self.bert( 2025-12-04T09:45:29.8416307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8416400Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8416699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8416797Z layer_outputs = layer_module( 2025-12-04T09:45:29.8417028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8417109Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8417351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8417428Z return func(*args, **kwargs) 2025-12-04T09:45:29.8417722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8417827Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8418077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8418146Z return func(*args, **kwargs) 2025-12-04T09:45:29.8418444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8418515Z self_outputs = self.self( 2025-12-04T09:45:29.8418756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8418831Z return func(*args, **kwargs) 2025-12-04T09:45:29.8419122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8419207Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8419211Z 2025-12-04T09:45:29.8419318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8419519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8419591Z res = mod(**inputs) 2025-12-04T09:45:29.8419882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8419949Z outputs = self.bert( 2025-12-04T09:45:29.8420246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8420319Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8420615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8420686Z layer_outputs = layer_module( 2025-12-04T09:45:29.8420911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8421002Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8421240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8421308Z return func(*args, **kwargs) 2025-12-04T09:45:29.8421604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8421686Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8421934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8422002Z return func(*args, **kwargs) 2025-12-04T09:45:29.8422299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8422395Z self_outputs = self.self( 2025-12-04T09:45:29.8422634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8422724Z return func(*args, **kwargs) 2025-12-04T09:45:29.8423028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8423108Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8423112Z 2025-12-04T09:45:29.8423198Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8423276Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8423375Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8423577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8423640Z res = mod(**inputs) 2025-12-04T09:45:29.8423931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8424016Z outputs = self.bert( 2025-12-04T09:45:29.8424297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8424376Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8424658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8424734Z layer_outputs = layer_module( 2025-12-04T09:45:29.8424953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8425029Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8425274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8425340Z return func(*args, **kwargs) 2025-12-04T09:45:29.8425626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8425717Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8425956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8426030Z return func(*args, **kwargs) 2025-12-04T09:45:29.8426313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8426438Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8426728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8426810Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8426813Z 2025-12-04T09:45:29.8426923Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8427118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8427182Z res = mod(**inputs) 2025-12-04T09:45:29.8427473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8427540Z outputs = self.bert( 2025-12-04T09:45:29.8427820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8427898Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8428179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8428257Z layer_outputs = layer_module( 2025-12-04T09:45:29.8428494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8428592Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8428835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8428900Z return func(*args, **kwargs) 2025-12-04T09:45:29.8429391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8429478Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8429747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8429835Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8430167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8430291Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8430583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8430666Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8430669Z 2025-12-04T09:45:29.8430781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8430976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8431041Z res = mod(**inputs) 2025-12-04T09:45:29.8431332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8431398Z outputs = self.bert( 2025-12-04T09:45:29.8431684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8431759Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8432043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8432123Z layer_outputs = layer_module( 2025-12-04T09:45:29.8432345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8432424Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8432670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8432738Z return func(*args, **kwargs) 2025-12-04T09:45:29.8433026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8433110Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8433390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8433480Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8433824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8433938Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8434228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8434342Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8434570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8434642Z return self.act(input) 2025-12-04T09:45:29.8434645Z 2025-12-04T09:45:29.8434749Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8434979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8435064Z res = mod(**inputs) 2025-12-04T09:45:29.8435376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8435462Z outputs = self.bert( 2025-12-04T09:45:29.8435755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8435840Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8436128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8436207Z layer_outputs = layer_module( 2025-12-04T09:45:29.8436444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8436561Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8436828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8436902Z return func(*args, **kwargs) 2025-12-04T09:45:29.8437208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8437308Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8437588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8437677Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8438016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8438161Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8438476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8438567Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8438570Z 2025-12-04T09:45:29.8438685Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8438898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8438968Z res = mod(**inputs) 2025-12-04T09:45:29.8439282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8439353Z outputs = self.bert( 2025-12-04T09:45:29.8439658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8439745Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8440053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8440136Z layer_outputs = layer_module( 2025-12-04T09:45:29.8440449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8440543Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8440816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8440892Z return func(*args, **kwargs) 2025-12-04T09:45:29.8441222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8441316Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8441629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8441752Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8442092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8442252Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8442571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.8442656Z return input_tensor + hidden_states 2025-12-04T09:45:29.8442661Z 2025-12-04T09:45:29.8442779Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8442992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8443062Z res = mod(**inputs) 2025-12-04T09:45:29.8443383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8443474Z outputs = self.bert( 2025-12-04T09:45:29.8443792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8443870Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8444182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8444265Z layer_outputs = layer_module( 2025-12-04T09:45:29.8444508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8444592Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8444860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8444937Z return func(*args, **kwargs) 2025-12-04T09:45:29.8445259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8445350Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8445615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8445697Z return func(*args, **kwargs) 2025-12-04T09:45:29.8446012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8446093Z self_outputs = self.self( 2025-12-04T09:45:29.8446354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8446425Z return func(*args, **kwargs) 2025-12-04T09:45:29.8446746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8446836Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8446840Z 2025-12-04T09:45:29.8446951Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8447378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8447457Z res = mod(**inputs) 2025-12-04T09:45:29.8447780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8447852Z outputs = self.bert( 2025-12-04T09:45:29.8448161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8448250Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8448605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8448717Z layer_outputs = layer_module( 2025-12-04T09:45:29.8448962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8449047Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8449340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8449417Z return func(*args, **kwargs) 2025-12-04T09:45:29.8449727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8449827Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8450089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8450171Z return func(*args, **kwargs) 2025-12-04T09:45:29.8450511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8450589Z self_outputs = self.self( 2025-12-04T09:45:29.8450859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8450935Z return func(*args, **kwargs) 2025-12-04T09:45:29.8451251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8451337Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8451341Z 2025-12-04T09:45:29.8451453Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8451671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8451741Z res = mod(**inputs) 2025-12-04T09:45:29.8452051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8452133Z outputs = self.bert( 2025-12-04T09:45:29.8452440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8452527Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8452831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8452906Z layer_outputs = layer_module( 2025-12-04T09:45:29.8453150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8453230Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8453473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8453553Z return func(*args, **kwargs) 2025-12-04T09:45:29.8453843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8453934Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8454180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8454249Z return func(*args, **kwargs) 2025-12-04T09:45:29.8454547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8454617Z self_outputs = self.self( 2025-12-04T09:45:29.8454869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8454939Z return func(*args, **kwargs) 2025-12-04T09:45:29.8455247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8455378Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8455381Z 2025-12-04T09:45:29.8455463Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8455544Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8455672Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8455877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8455949Z res = mod(**inputs) 2025-12-04T09:45:29.8456253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8456322Z outputs = self.bert( 2025-12-04T09:45:29.8456624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8456718Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8457026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8457098Z layer_outputs = layer_module( 2025-12-04T09:45:29.8457331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8457419Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8457668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8457736Z return func(*args, **kwargs) 2025-12-04T09:45:29.8458044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8458126Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8458386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8458457Z return func(*args, **kwargs) 2025-12-04T09:45:29.8458759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8458898Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8459200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8459289Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8459293Z 2025-12-04T09:45:29.8459398Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8459605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8459678Z res = mod(**inputs) 2025-12-04T09:45:29.8459984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8460055Z outputs = self.bert( 2025-12-04T09:45:29.8460364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8460438Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8460741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8460812Z layer_outputs = layer_module( 2025-12-04T09:45:29.8461046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8461133Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8461384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8461471Z return func(*args, **kwargs) 2025-12-04T09:45:29.8461787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8461873Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8462163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8462242Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8462562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8462677Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8462965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8463059Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8463079Z 2025-12-04T09:45:29.8463186Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8463387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8463463Z res = mod(**inputs) 2025-12-04T09:45:29.8463759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8463835Z outputs = self.bert( 2025-12-04T09:45:29.8464124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8464197Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8464496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8464567Z layer_outputs = layer_module( 2025-12-04T09:45:29.8464795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8464884Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8465128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8465207Z return func(*args, **kwargs) 2025-12-04T09:45:29.8465498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8465584Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8465858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8465933Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8466260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8466369Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8466662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8466795Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8467011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8467083Z return self.act(input) 2025-12-04T09:45:29.8467086Z 2025-12-04T09:45:29.8467198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8467398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8467472Z res = mod(**inputs) 2025-12-04T09:45:29.8467791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8467876Z outputs = self.bert( 2025-12-04T09:45:29.8468177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8468252Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8468563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8468636Z layer_outputs = layer_module( 2025-12-04T09:45:29.8468862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8468950Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8469197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8469267Z return func(*args, **kwargs) 2025-12-04T09:45:29.8469590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8469678Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8469955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8470031Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8470354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8470497Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8470795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8470884Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8470888Z 2025-12-04T09:45:29.8470991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8471191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8471262Z res = mod(**inputs) 2025-12-04T09:45:29.8471549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8471615Z outputs = self.bert( 2025-12-04T09:45:29.8471906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8471978Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8472268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8472336Z layer_outputs = layer_module( 2025-12-04T09:45:29.8472557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8472645Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8472883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8472958Z return func(*args, **kwargs) 2025-12-04T09:45:29.8473239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8473319Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8473564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8473630Z return func(*args, **kwargs) 2025-12-04T09:45:29.8473915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8474009Z self_outputs = self.self( 2025-12-04T09:45:29.8474269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8474344Z return func(*args, **kwargs) 2025-12-04T09:45:29.8474647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8474733Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8474737Z 2025-12-04T09:45:29.8474849Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8475052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8475124Z res = mod(**inputs) 2025-12-04T09:45:29.8475421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8475489Z outputs = self.bert( 2025-12-04T09:45:29.8475805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8475883Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8476177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8476258Z layer_outputs = layer_module( 2025-12-04T09:45:29.8476488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8476576Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8476822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8476891Z return func(*args, **kwargs) 2025-12-04T09:45:29.8477198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8477287Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8477551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8477631Z return func(*args, **kwargs) 2025-12-04T09:45:29.8477942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8478023Z self_outputs = self.self( 2025-12-04T09:45:29.8478281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8478354Z return func(*args, **kwargs) 2025-12-04T09:45:29.8478671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8478757Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8478763Z 2025-12-04T09:45:29.8478886Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8479103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8479172Z res = mod(**inputs) 2025-12-04T09:45:29.8479496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8479569Z outputs = self.bert( 2025-12-04T09:45:29.8479879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8479966Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8480274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8480428Z layer_outputs = layer_module( 2025-12-04T09:45:29.8480709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8480817Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8481092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8481188Z return func(*args, **kwargs) 2025-12-04T09:45:29.8481526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8481618Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8481887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8481976Z return func(*args, **kwargs) 2025-12-04T09:45:29.8482292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8482384Z self_outputs = self.self( 2025-12-04T09:45:29.8482641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8482711Z return func(*args, **kwargs) 2025-12-04T09:45:29.8483011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8483096Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8483100Z 2025-12-04T09:45:29.8483181Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8483270Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8483374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8483575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8483649Z res = mod(**inputs) 2025-12-04T09:45:29.8483944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8484020Z outputs = self.bert( 2025-12-04T09:45:29.8484308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8484384Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8484680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8484750Z layer_outputs = layer_module( 2025-12-04T09:45:29.8484980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8485061Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8485303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8485382Z return func(*args, **kwargs) 2025-12-04T09:45:29.8485670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8485754Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8486010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8486079Z return func(*args, **kwargs) 2025-12-04T09:45:29.8486379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8486510Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8486800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8486900Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8486923Z 2025-12-04T09:45:29.8487044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8487249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8487314Z res = mod(**inputs) 2025-12-04T09:45:29.8487622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8487701Z outputs = self.bert( 2025-12-04T09:45:29.8487993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8488067Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8488368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8488439Z layer_outputs = layer_module( 2025-12-04T09:45:29.8488726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8488807Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8489052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8489131Z return func(*args, **kwargs) 2025-12-04T09:45:29.8489423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8489517Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8489787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8489865Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8490195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8490303Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8490592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8490683Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8490688Z 2025-12-04T09:45:29.8490793Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8491001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8491066Z res = mod(**inputs) 2025-12-04T09:45:29.8491359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8491435Z outputs = self.bert( 2025-12-04T09:45:29.8491725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8491806Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8492099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8492169Z layer_outputs = layer_module( 2025-12-04T09:45:29.8492405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8492485Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8492729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8492805Z return func(*args, **kwargs) 2025-12-04T09:45:29.8493096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8493187Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8493473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8493568Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8493909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8494016Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8494312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8494428Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8494646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8494725Z return self.act(input) 2025-12-04T09:45:29.8494729Z 2025-12-04T09:45:29.8494853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8495057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8495130Z res = mod(**inputs) 2025-12-04T09:45:29.8495428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8495505Z outputs = self.bert( 2025-12-04T09:45:29.8495803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8495876Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8496177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8496249Z layer_outputs = layer_module( 2025-12-04T09:45:29.8496487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8496571Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8496816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8496895Z return func(*args, **kwargs) 2025-12-04T09:45:29.8497188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8497274Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8497549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8497628Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8497956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8498096Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8498395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8498489Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8498492Z 2025-12-04T09:45:29.8498599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8498807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8498872Z res = mod(**inputs) 2025-12-04T09:45:29.8499169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8499246Z outputs = self.bert( 2025-12-04T09:45:29.8499538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8499634Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8499946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8500017Z layer_outputs = layer_module( 2025-12-04T09:45:29.8500266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8500348Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8500594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8500673Z return func(*args, **kwargs) 2025-12-04T09:45:29.8500964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8501055Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8501323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8501420Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8501747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8501883Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8502174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.8502259Z return input_tensor + hidden_states 2025-12-04T09:45:29.8502263Z 2025-12-04T09:45:29.8502369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8502580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8502646Z res = mod(**inputs) 2025-12-04T09:45:29.8502944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8503022Z outputs = self.bert( 2025-12-04T09:45:29.8503316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8503398Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8503688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8503759Z layer_outputs = layer_module( 2025-12-04T09:45:29.8503995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8504076Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8504325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8504406Z return func(*args, **kwargs) 2025-12-04T09:45:29.8504700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8504791Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8505037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8505107Z return func(*args, **kwargs) 2025-12-04T09:45:29.8505403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8505475Z self_outputs = self.self( 2025-12-04T09:45:29.8505728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8505797Z return func(*args, **kwargs) 2025-12-04T09:45:29.8506115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8506229Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8506233Z 2025-12-04T09:45:29.8506342Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8506574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8506655Z res = mod(**inputs) 2025-12-04T09:45:29.8506967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8507046Z outputs = self.bert( 2025-12-04T09:45:29.8507357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8507432Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8507728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8507819Z layer_outputs = layer_module( 2025-12-04T09:45:29.8508049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8508129Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8508370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8508449Z return func(*args, **kwargs) 2025-12-04T09:45:29.8508753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8508840Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8509104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8509179Z return func(*args, **kwargs) 2025-12-04T09:45:29.8509492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8509566Z self_outputs = self.self( 2025-12-04T09:45:29.8509824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8509903Z return func(*args, **kwargs) 2025-12-04T09:45:29.8510207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8510299Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8510306Z 2025-12-04T09:45:29.8510416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8510628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8510705Z res = mod(**inputs) 2025-12-04T09:45:29.8511018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8511092Z outputs = self.bert( 2025-12-04T09:45:29.8511406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8511485Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8511798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8511874Z layer_outputs = layer_module( 2025-12-04T09:45:29.8512112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8512204Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8512481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8512573Z return func(*args, **kwargs) 2025-12-04T09:45:29.8512894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8512982Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8513286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8513361Z return func(*args, **kwargs) 2025-12-04T09:45:29.8513670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8513751Z self_outputs = self.self( 2025-12-04T09:45:29.8514010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8514090Z return func(*args, **kwargs) 2025-12-04T09:45:29.8514414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8514501Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8514505Z 2025-12-04T09:45:29.8514598Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8514683Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8514795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8515016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8515085Z res = mod(**inputs) 2025-12-04T09:45:29.8515402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8515473Z outputs = self.bert( 2025-12-04T09:45:29.8515782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8515870Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8516174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8516255Z layer_outputs = layer_module( 2025-12-04T09:45:29.8516493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8516575Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8516837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8516910Z return func(*args, **kwargs) 2025-12-04T09:45:29.8517215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8517307Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8517568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8517647Z return func(*args, **kwargs) 2025-12-04T09:45:29.8517954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8518089Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8518405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8518492Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8518495Z 2025-12-04T09:45:29.8518612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8518823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8518894Z res = mod(**inputs) 2025-12-04T09:45:29.8519229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8519321Z outputs = self.bert( 2025-12-04T09:45:29.8519642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8519732Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8520043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8520125Z layer_outputs = layer_module( 2025-12-04T09:45:29.8520438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8520528Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8520805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8520910Z return func(*args, **kwargs) 2025-12-04T09:45:29.8521229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8521332Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8521628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8521717Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8522056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8522169Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8522491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8522581Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8522587Z 2025-12-04T09:45:29.8522706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8522920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8522990Z res = mod(**inputs) 2025-12-04T09:45:29.8523310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8523380Z outputs = self.bert( 2025-12-04T09:45:29.8523690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8523767Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8524075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8524162Z layer_outputs = layer_module( 2025-12-04T09:45:29.8524404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8524486Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8524755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8524825Z return func(*args, **kwargs) 2025-12-04T09:45:29.8525120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8525204Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8525471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8525554Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8525887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8526017Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8526306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8526437Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8526663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8526735Z return self.act(input) 2025-12-04T09:45:29.8526738Z 2025-12-04T09:45:29.8526842Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8527050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8527114Z res = mod(**inputs) 2025-12-04T09:45:29.8527413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8527505Z outputs = self.bert( 2025-12-04T09:45:29.8527805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8527887Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8528187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8528267Z layer_outputs = layer_module( 2025-12-04T09:45:29.8528496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8528577Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8528837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8528909Z return func(*args, **kwargs) 2025-12-04T09:45:29.8529209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8529301Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8529574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8529660Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8529988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8530123Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8530429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8530514Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8530520Z 2025-12-04T09:45:29.8530634Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8530838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8530906Z res = mod(**inputs) 2025-12-04T09:45:29.8531222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8531292Z outputs = self.bert( 2025-12-04T09:45:29.8531596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8531679Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8531986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8532066Z layer_outputs = layer_module( 2025-12-04T09:45:29.8532325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8532435Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8532693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8532781Z return func(*args, **kwargs) 2025-12-04T09:45:29.8533090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8533177Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8533427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8533504Z return func(*args, **kwargs) 2025-12-04T09:45:29.8533803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8533895Z self_outputs = self.self( 2025-12-04T09:45:29.8534153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8534223Z return func(*args, **kwargs) 2025-12-04T09:45:29.8534528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:45:29.8534614Z query_layer = self.query(hidden_states) 2025-12-04T09:45:29.8534618Z 2025-12-04T09:45:29.8534727Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8534941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8535008Z res = mod(**inputs) 2025-12-04T09:45:29.8535309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8535389Z outputs = self.bert( 2025-12-04T09:45:29.8535687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8535770Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8536069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8536142Z layer_outputs = layer_module( 2025-12-04T09:45:29.8536381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8536460Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8536720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8536791Z return func(*args, **kwargs) 2025-12-04T09:45:29.8537090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8537184Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8537434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8537507Z return func(*args, **kwargs) 2025-12-04T09:45:29.8537817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8537889Z self_outputs = self.self( 2025-12-04T09:45:29.8538147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8538217Z return func(*args, **kwargs) 2025-12-04T09:45:29.8538513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:45:29.8538626Z key_layer = self.key(current_states) 2025-12-04T09:45:29.8538656Z 2025-12-04T09:45:29.8538766Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8538981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8539048Z res = mod(**inputs) 2025-12-04T09:45:29.8539373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8539455Z outputs = self.bert( 2025-12-04T09:45:29.8539769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8539857Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8540164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8540257Z layer_outputs = layer_module( 2025-12-04T09:45:29.8540502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8540581Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8540826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8540902Z return func(*args, **kwargs) 2025-12-04T09:45:29.8541192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8541282Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8541523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8541592Z return func(*args, **kwargs) 2025-12-04T09:45:29.8541893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:45:29.8541965Z self_outputs = self.self( 2025-12-04T09:45:29.8542211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8542286Z return func(*args, **kwargs) 2025-12-04T09:45:29.8542576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:45:29.8542665Z value_layer = self.value(current_states) 2025-12-04T09:45:29.8542668Z 2025-12-04T09:45:29.8542749Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8542829Z cudagraph partition due to non gpu ops 2025-12-04T09:45:29.8542942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8543141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8543205Z res = mod(**inputs) 2025-12-04T09:45:29.8543510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8543580Z outputs = self.bert( 2025-12-04T09:45:29.8543887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8543961Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8544253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8544333Z layer_outputs = layer_module( 2025-12-04T09:45:29.8544560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8544649Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8544910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8545001Z return func(*args, **kwargs) 2025-12-04T09:45:29.8545313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:45:29.8545398Z self_attention_outputs = self.attention( 2025-12-04T09:45:29.8545666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8545746Z return func(*args, **kwargs) 2025-12-04T09:45:29.8546035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:45:29.8546169Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:29.8546457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:45:29.8546562Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8546566Z 2025-12-04T09:45:29.8546678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8546879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8546954Z res = mod(**inputs) 2025-12-04T09:45:29.8547364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8547439Z outputs = self.bert( 2025-12-04T09:45:29.8547746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8547823Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8548116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8548199Z layer_outputs = layer_module( 2025-12-04T09:45:29.8548430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8548520Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8548770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8548840Z return func(*args, **kwargs) 2025-12-04T09:45:29.8549145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8549230Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8549510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8549588Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8549916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8550033Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8550330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:45:29.8550417Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8550429Z 2025-12-04T09:45:29.8550535Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8550737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8550811Z res = mod(**inputs) 2025-12-04T09:45:29.8551107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8551173Z outputs = self.bert( 2025-12-04T09:45:29.8551515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8551614Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8551923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8552013Z layer_outputs = layer_module( 2025-12-04T09:45:29.8552234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8552322Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8552563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8552641Z return func(*args, **kwargs) 2025-12-04T09:45:29.8552926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8553035Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8553298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8553372Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8553679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:45:29.8553789Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:45:29.8554074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:45:29.8554194Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:29.8554407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:29.8554478Z return self.act(input) 2025-12-04T09:45:29.8554483Z 2025-12-04T09:45:29.8554593Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8554788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8554851Z res = mod(**inputs) 2025-12-04T09:45:29.8555155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8555220Z outputs = self.bert( 2025-12-04T09:45:29.8555501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8555571Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8555843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8555916Z layer_outputs = layer_module( 2025-12-04T09:45:29.8556133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8556215Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8556459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8556528Z return func(*args, **kwargs) 2025-12-04T09:45:29.8556817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8556900Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8557160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8557241Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8557583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8557747Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8558036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:45:29.8558119Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8558141Z 2025-12-04T09:45:29.8558256Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8558458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8558531Z res = mod(**inputs) 2025-12-04T09:45:29.8558825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:45:29.8558893Z outputs = self.bert( 2025-12-04T09:45:29.8559190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:45:29.8559283Z encoder_outputs = self.encoder( 2025-12-04T09:45:29.8559577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:45:29.8559655Z layer_outputs = layer_module( 2025-12-04T09:45:29.8559887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:29.8559978Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:29.8560229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:29.8560340Z return func(*args, **kwargs) 2025-12-04T09:45:29.8595554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:45:29.8595810Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:29.8596120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:29.8596215Z return forward_fn(*input_tensors) 2025-12-04T09:45:29.8596612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:45:29.8596807Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:29.8597127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:45:29.8597209Z return input_tensor + hidden_states 2025-12-04T09:45:29.8597218Z 2025-12-04T09:45:29.8597386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8597601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8597685Z res = mod(**inputs) 2025-12-04T09:45:29.8597978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1074, in forward 2025-12-04T09:45:29.8598077Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:45:29.8598373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 644, in forward 2025-12-04T09:45:29.8598490Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:45:29.8598779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 632, in forward 2025-12-04T09:45:29.8598885Z hidden_states = self.transform(hidden_states) 2025-12-04T09:45:29.8599218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 607, in forward 2025-12-04T09:45:29.8599420Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:29.8599428Z 2025-12-04T09:45:29.8599593Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8599820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8599904Z res = mod(**inputs) 2025-12-04T09:45:29.8600258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1074, in forward 2025-12-04T09:45:29.8600576Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:45:29.8600897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 644, in forward 2025-12-04T09:45:29.8601017Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:45:29.8601336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 633, in forward 2025-12-04T09:45:29.8601500Z hidden_states = self.decoder(hidden_states) 2025-12-04T09:45:29.8601506Z 2025-12-04T09:45:29.8601628Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:29.8601847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:29.8601919Z res = mod(**inputs) 2025-12-04T09:45:29.8602234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1078, in forward 2025-12-04T09:45:29.8602310Z lm_loss = self.loss_function( 2025-12-04T09:45:29.8602558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:45:29.8602742Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:45:29.8602994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:45:29.8603199Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:45:29.8603204Z 2025-12-04T09:45:41.3007583Z Compilation time (from dynamo_timed): 25.087344874 2025-12-04T09:45:41.3039474Z pass 2025-12-04T09:45:41.3039894Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:41.3041066Z TIMING: _recursive_pre_grad_passes:0.01099 _recursive_joint_graph_passes:0.77435 _recursive_post_grad_passes:0.107 async_compile.wait:0.84799 code_gen:11.02791 inductor_compile:13.43515 backend_compile:19.47766 gc:0.00021 entire_frame_compile:25.08734 total_wall_time:25.08734 2025-12-04T09:45:41.3042069Z STATS: call_* op count: 723 | FakeTensorMode.__torch_dispatch__:16128 | FakeTensor.__torch_dispatch__:8250 | ProxyTorchDispatchMode.__torch_dispatch__:4916 2025-12-04T09:45:41.3042586Z Dynamo produced 1 graphs covering 723 ops with 0 graph breaks (0 unique) 2025-12-04T09:45:44.0042372Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:45:44.0043945Z import pynvml # type: ignore[import] 2025-12-04T09:45:47.5550221Z 2025-12-04T09:45:48.0529918Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:45:48.0532375Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:45:48.0598964Z cpu eval MobileBertForMaskedLM 2025-12-04T09:45:48.3418205Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:48.5051137Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:48.8656103Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:16.5092217Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5092648Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5092986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5093421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5093800Z res = mod(**inputs) 2025-12-04T09:46:16.5094386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5094878Z outputs = self.mobilebert( 2025-12-04T09:46:16.5095325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-12-04T09:46:16.5095830Z embedding_output = self.embeddings( 2025-12-04T09:46:16.5096439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-12-04T09:46:16.5096905Z inputs_embeds = torch.cat( 2025-12-04T09:46:16.5097098Z 2025-12-04T09:46:16.5097227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5097627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5097983Z res = mod(**inputs) 2025-12-04T09:46:16.5098443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T09:46:16.5098914Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:46:16.5099387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T09:46:16.5099874Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:46:16.5100370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-12-04T09:46:16.5100976Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-12-04T09:46:16.5101254Z 2025-12-04T09:46:16.5101370Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5101759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5102115Z res = mod(**inputs) 2025-12-04T09:46:16.5102544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5102991Z outputs = self.mobilebert( 2025-12-04T09:46:16.5103428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-12-04T09:46:16.5103886Z embedding_output = self.embeddings( 2025-12-04T09:46:16.5104338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-12-04T09:46:16.5104847Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-12-04T09:46:16.5105045Z 2025-12-04T09:46:16.5105157Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5105538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5105879Z res = mod(**inputs) 2025-12-04T09:46:16.5106302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5106754Z outputs = self.mobilebert( 2025-12-04T09:46:16.5107182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-12-04T09:46:16.5107646Z embedding_output = self.embeddings( 2025-12-04T09:46:16.5108118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-12-04T09:46:16.5108619Z embeddings = self.LayerNorm(embeddings) 2025-12-04T09:46:16.5109120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5109613Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5109779Z 2025-12-04T09:46:16.5109908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5110295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5110658Z res = mod(**inputs) 2025-12-04T09:46:16.5111096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5111562Z outputs = self.mobilebert( 2025-12-04T09:46:16.5112009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5112468Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5112969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5113455Z layer_outputs = layer_module( 2025-12-04T09:46:16.5113913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5114485Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5115056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5115572Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5116069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5116547Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5116709Z 2025-12-04T09:46:16.5116827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5117230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5117605Z res = mod(**inputs) 2025-12-04T09:46:16.5118048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5118515Z outputs = self.mobilebert( 2025-12-04T09:46:16.5118965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5119429Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5119892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5120513Z layer_outputs = layer_module( 2025-12-04T09:46:16.5120982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5121458Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5121954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5122422Z self_outputs = self.self( 2025-12-04T09:46:16.5122867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.5123334Z self.value(value_tensor) 2025-12-04T09:46:16.5123457Z 2025-12-04T09:46:16.5123579Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5123974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5124323Z res = mod(**inputs) 2025-12-04T09:46:16.5124781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5125257Z outputs = self.mobilebert( 2025-12-04T09:46:16.5125693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5126159Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5126631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5127091Z layer_outputs = layer_module( 2025-12-04T09:46:16.5127538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5128127Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5128695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.5129224Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.5129724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5130195Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5130353Z 2025-12-04T09:46:16.5130479Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5130876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5131239Z res = mod(**inputs) 2025-12-04T09:46:16.5131667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5132125Z outputs = self.mobilebert( 2025-12-04T09:46:16.5132567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5133033Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5133490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5133957Z layer_outputs = layer_module( 2025-12-04T09:46:16.5134403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5134954Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5135513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5136013Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5136512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.5137004Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.5137486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5137967Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5138139Z 2025-12-04T09:46:16.5138259Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5138663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5139027Z res = mod(**inputs) 2025-12-04T09:46:16.5139455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5139915Z outputs = self.mobilebert( 2025-12-04T09:46:16.5140362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5140848Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5141321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5141931Z layer_outputs = layer_module( 2025-12-04T09:46:16.5142455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5142966Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5143438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5143890Z self_outputs = self.self( 2025-12-04T09:46:16.5144332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.5144779Z self.query(query_tensor) 2025-12-04T09:46:16.5144915Z 2025-12-04T09:46:16.5145062Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5145464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5145823Z res = mod(**inputs) 2025-12-04T09:46:16.5146251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5146693Z outputs = self.mobilebert( 2025-12-04T09:46:16.5147311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5147737Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5148154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5148573Z layer_outputs = layer_module( 2025-12-04T09:46:16.5148986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5149417Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5149862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5150315Z self_outputs = self.self( 2025-12-04T09:46:16.5150736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.5151176Z self.key(key_tensor) 2025-12-04T09:46:16.5151300Z 2025-12-04T09:46:16.5151390Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5151629Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5151879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5152267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5152617Z res = mod(**inputs) 2025-12-04T09:46:16.5153039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5153462Z outputs = self.mobilebert( 2025-12-04T09:46:16.5153870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5154294Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5154704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5155143Z layer_outputs = layer_module( 2025-12-04T09:46:16.5155576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5156032Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5156554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5157100Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5157613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.5158136Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5158294Z 2025-12-04T09:46:16.5158410Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5158807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5159187Z res = mod(**inputs) 2025-12-04T09:46:16.5159617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5160082Z outputs = self.mobilebert( 2025-12-04T09:46:16.5160611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5161113Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5161565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5162037Z layer_outputs = layer_module( 2025-12-04T09:46:16.5162449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5162923Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5163383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5163904Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5164416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.5164932Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5165460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5165951Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5166113Z 2025-12-04T09:46:16.5166236Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5166623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5166979Z res = mod(**inputs) 2025-12-04T09:46:16.5167418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5167873Z outputs = self.mobilebert( 2025-12-04T09:46:16.5168307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5168765Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5169220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5169667Z layer_outputs = layer_module( 2025-12-04T09:46:16.5170120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5170605Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5171089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5171593Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5172078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5172557Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5172722Z 2025-12-04T09:46:16.5173339Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5173699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5174032Z res = mod(**inputs) 2025-12-04T09:46:16.5174464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5174905Z outputs = self.mobilebert( 2025-12-04T09:46:16.5175312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5175732Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5176163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5176602Z layer_outputs = layer_module( 2025-12-04T09:46:16.5177037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5177499Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5177972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5178422Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5178878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5179345Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5179525Z 2025-12-04T09:46:16.5179644Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5180018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5180444Z res = mod(**inputs) 2025-12-04T09:46:16.5180865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5181296Z outputs = self.mobilebert( 2025-12-04T09:46:16.5181703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5182122Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5182539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5182947Z layer_outputs = layer_module( 2025-12-04T09:46:16.5183362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5183815Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5184282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5184784Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5185290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5185747Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5185896Z 2025-12-04T09:46:16.5186007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5186385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5186712Z res = mod(**inputs) 2025-12-04T09:46:16.5187101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5187579Z outputs = self.mobilebert( 2025-12-04T09:46:16.5188036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5188490Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5188897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5189357Z layer_outputs = layer_module( 2025-12-04T09:46:16.5189760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5190198Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5190637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5191104Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5191567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5192057Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5192523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5192962Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5193123Z 2025-12-04T09:46:16.5193234Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5193621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5193974Z res = mod(**inputs) 2025-12-04T09:46:16.5194377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5194818Z outputs = self.mobilebert( 2025-12-04T09:46:16.5195251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5195693Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5196122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5196559Z layer_outputs = layer_module( 2025-12-04T09:46:16.5196988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5197442Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5197902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5198390Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5198873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5199323Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5199483Z 2025-12-04T09:46:16.5199592Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5199975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5200411Z res = mod(**inputs) 2025-12-04T09:46:16.5200849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5201312Z outputs = self.mobilebert( 2025-12-04T09:46:16.5201759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5202233Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5202677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5203147Z layer_outputs = layer_module( 2025-12-04T09:46:16.5203629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5204094Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5204592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5205075Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5205561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5206040Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5206225Z 2025-12-04T09:46:16.5206336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5206723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5207103Z res = mod(**inputs) 2025-12-04T09:46:16.5207517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5207970Z outputs = self.mobilebert( 2025-12-04T09:46:16.5208409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5208850Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5209300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5209757Z layer_outputs = layer_module( 2025-12-04T09:46:16.5210197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5210663Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5211141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5211654Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5212156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5212607Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5212765Z 2025-12-04T09:46:16.5212876Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5213265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5213613Z res = mod(**inputs) 2025-12-04T09:46:16.5214043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5214486Z outputs = self.mobilebert( 2025-12-04T09:46:16.5214921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5215366Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5215808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5216258Z layer_outputs = layer_module( 2025-12-04T09:46:16.5216697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5217163Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5217629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5218131Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5219389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5219942Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5220451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5220953Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5221116Z 2025-12-04T09:46:16.5221230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5221618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5221966Z res = mod(**inputs) 2025-12-04T09:46:16.5222392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5222839Z outputs = self.mobilebert( 2025-12-04T09:46:16.5223272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5223744Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5224195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5224690Z layer_outputs = layer_module( 2025-12-04T09:46:16.5225133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5225607Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5226082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5226580Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5227086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5227568Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5227723Z 2025-12-04T09:46:16.5227838Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5228251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5228599Z res = mod(**inputs) 2025-12-04T09:46:16.5229037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5229483Z outputs = self.mobilebert( 2025-12-04T09:46:16.5229923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5230378Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5230818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5231276Z layer_outputs = layer_module( 2025-12-04T09:46:16.5231721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5232196Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5232667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5233178Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5233670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5234169Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5234349Z 2025-12-04T09:46:16.5234462Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5234878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5235260Z res = mod(**inputs) 2025-12-04T09:46:16.5235683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5236147Z outputs = self.mobilebert( 2025-12-04T09:46:16.5236617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5237083Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5237531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5237986Z layer_outputs = layer_module( 2025-12-04T09:46:16.5238436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5238913Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5239426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5239948Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5240572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5241051Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5241217Z 2025-12-04T09:46:16.5241335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5241737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5242102Z res = mod(**inputs) 2025-12-04T09:46:16.5242525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5242991Z outputs = self.mobilebert( 2025-12-04T09:46:16.5243442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5243906Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5244358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5244809Z layer_outputs = layer_module( 2025-12-04T09:46:16.5245247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5245706Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5246175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5246674Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5247326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5247832Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5248338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5248819Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5248979Z 2025-12-04T09:46:16.5249098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5249479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5249830Z res = mod(**inputs) 2025-12-04T09:46:16.5250251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5250694Z outputs = self.mobilebert( 2025-12-04T09:46:16.5251164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5251637Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5252071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5252527Z layer_outputs = layer_module( 2025-12-04T09:46:16.5252966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5253473Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5253968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5254429Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5254590Z 2025-12-04T09:46:16.5254701Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5255120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5255471Z res = mod(**inputs) 2025-12-04T09:46:16.5255876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5256317Z outputs = self.mobilebert( 2025-12-04T09:46:16.5256749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5257186Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5257628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5258064Z layer_outputs = layer_module( 2025-12-04T09:46:16.5258497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5258984Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5259476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5259972Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5260150Z 2025-12-04T09:46:16.5260268Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5260644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5260990Z res = mod(**inputs) 2025-12-04T09:46:16.5261404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5261837Z outputs = self.mobilebert( 2025-12-04T09:46:16.5262266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5262711Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5263153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5263593Z layer_outputs = layer_module( 2025-12-04T09:46:16.5264031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5264579Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5265128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.5265597Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.5265780Z 2025-12-04T09:46:16.5265887Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5266289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5266644Z res = mod(**inputs) 2025-12-04T09:46:16.5267057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5267483Z outputs = self.mobilebert( 2025-12-04T09:46:16.5267908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5268323Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5268765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5269204Z layer_outputs = layer_module( 2025-12-04T09:46:16.5269648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5270194Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5270758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.5271258Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.5271753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5272227Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5272396Z 2025-12-04T09:46:16.5272512Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5272908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5273253Z res = mod(**inputs) 2025-12-04T09:46:16.5273669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5274115Z outputs = self.mobilebert( 2025-12-04T09:46:16.5274555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5275003Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5275442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5275887Z layer_outputs = layer_module( 2025-12-04T09:46:16.5276314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5276846Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5277381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5277880Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5278374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.5278846Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5279007Z 2025-12-04T09:46:16.5279122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5279516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5279865Z res = mod(**inputs) 2025-12-04T09:46:16.5280290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5280837Z outputs = self.mobilebert( 2025-12-04T09:46:16.5281283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5281761Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5282241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5282711Z layer_outputs = layer_module( 2025-12-04T09:46:16.5283184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5283740Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5284307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5284820Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5285324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.5285859Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5286400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5286889Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5287053Z 2025-12-04T09:46:16.5287170Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5287564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5287938Z res = mod(**inputs) 2025-12-04T09:46:16.5288359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5288822Z outputs = self.mobilebert( 2025-12-04T09:46:16.5289273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5289742Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5290189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5290648Z layer_outputs = layer_module( 2025-12-04T09:46:16.5291092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5291634Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5292164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5292649Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5293129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5293596Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5293755Z 2025-12-04T09:46:16.5293871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5294265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5294613Z res = mod(**inputs) 2025-12-04T09:46:16.5295020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5295463Z outputs = self.mobilebert( 2025-12-04T09:46:16.5295890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5296331Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5296760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5297201Z layer_outputs = layer_module( 2025-12-04T09:46:16.5297657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5298141Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5298597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5299056Z self_outputs = self.self( 2025-12-04T09:46:16.5299487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.5299921Z self.value(value_tensor) 2025-12-04T09:46:16.5300053Z 2025-12-04T09:46:16.5300167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5300554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5300898Z res = mod(**inputs) 2025-12-04T09:46:16.5301309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5301775Z outputs = self.mobilebert( 2025-12-04T09:46:16.5302203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5302642Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5303083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5303526Z layer_outputs = layer_module( 2025-12-04T09:46:16.5303969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5304499Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5305041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.5305502Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.5305959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5306384Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5306532Z 2025-12-04T09:46:16.5306639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5307001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5307331Z res = mod(**inputs) 2025-12-04T09:46:16.5307716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5308131Z outputs = self.mobilebert( 2025-12-04T09:46:16.5308535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5308951Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5309375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5309815Z layer_outputs = layer_module( 2025-12-04T09:46:16.5310251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5310778Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5311314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5311801Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5312280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.5312766Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.5313247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5313691Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5313841Z 2025-12-04T09:46:16.5313964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5314330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5314663Z res = mod(**inputs) 2025-12-04T09:46:16.5315076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5315529Z outputs = self.mobilebert( 2025-12-04T09:46:16.5315961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5316444Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5316896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5317331Z layer_outputs = layer_module( 2025-12-04T09:46:16.5317767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5318225Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5318676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5319121Z self_outputs = self.self( 2025-12-04T09:46:16.5319547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.5319985Z self.query(query_tensor) 2025-12-04T09:46:16.5320109Z 2025-12-04T09:46:16.5320225Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5320721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5321083Z res = mod(**inputs) 2025-12-04T09:46:16.5321513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5321969Z outputs = self.mobilebert( 2025-12-04T09:46:16.5322412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5322861Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5323297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5323745Z layer_outputs = layer_module( 2025-12-04T09:46:16.5324189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5324672Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5325147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5325608Z self_outputs = self.self( 2025-12-04T09:46:16.5326041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.5326496Z self.key(key_tensor) 2025-12-04T09:46:16.5326616Z 2025-12-04T09:46:16.5326708Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5326949Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5327209Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5327593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5327947Z res = mod(**inputs) 2025-12-04T09:46:16.5328403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5328875Z outputs = self.mobilebert( 2025-12-04T09:46:16.5329331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5329826Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5330280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5330735Z layer_outputs = layer_module( 2025-12-04T09:46:16.5331181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5331651Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5332115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5332629Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5333132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.5333623Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5333774Z 2025-12-04T09:46:16.5333897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5334280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5334629Z res = mod(**inputs) 2025-12-04T09:46:16.5335051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5335497Z outputs = self.mobilebert( 2025-12-04T09:46:16.5335930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5336376Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5336815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5337254Z layer_outputs = layer_module( 2025-12-04T09:46:16.5337687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5338148Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5338599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5339085Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5339586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.5340116Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5340640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5341106Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5341271Z 2025-12-04T09:46:16.5341389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5341775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5342127Z res = mod(**inputs) 2025-12-04T09:46:16.5342555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5343011Z outputs = self.mobilebert( 2025-12-04T09:46:16.5343457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5343939Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5344414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5344869Z layer_outputs = layer_module( 2025-12-04T09:46:16.5345323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5345816Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5346295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5346792Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5347720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5348241Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5348458Z 2025-12-04T09:46:16.5348573Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5348968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5349353Z res = mod(**inputs) 2025-12-04T09:46:16.5349781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5350257Z outputs = self.mobilebert( 2025-12-04T09:46:16.5350714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5351200Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5351651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5352133Z layer_outputs = layer_module( 2025-12-04T09:46:16.5352576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5353063Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5353567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5354094Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5354604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5355100Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5355278Z 2025-12-04T09:46:16.5355396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5355772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5356123Z res = mod(**inputs) 2025-12-04T09:46:16.5356549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5357010Z outputs = self.mobilebert( 2025-12-04T09:46:16.5357441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5357895Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5358343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5358778Z layer_outputs = layer_module( 2025-12-04T09:46:16.5359201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5359684Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5360193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5360815Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5361345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5361853Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5362012Z 2025-12-04T09:46:16.5362134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5362537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5362887Z res = mod(**inputs) 2025-12-04T09:46:16.5363304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5363746Z outputs = self.mobilebert( 2025-12-04T09:46:16.5364169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5364638Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5365074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5365508Z layer_outputs = layer_module( 2025-12-04T09:46:16.5365941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5366404Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5366923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5367424Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5367937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5368437Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5368937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5369396Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5369568Z 2025-12-04T09:46:16.5369681Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5370071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5370421Z res = mod(**inputs) 2025-12-04T09:46:16.5370835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5371285Z outputs = self.mobilebert( 2025-12-04T09:46:16.5371725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5372171Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5372615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5373068Z layer_outputs = layer_module( 2025-12-04T09:46:16.5373509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5373981Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5374456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5374947Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5375449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5375925Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5376100Z 2025-12-04T09:46:16.5376210Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5376593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5376932Z res = mod(**inputs) 2025-12-04T09:46:16.5377378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5377828Z outputs = self.mobilebert( 2025-12-04T09:46:16.5378254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5378668Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5379079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5379496Z layer_outputs = layer_module( 2025-12-04T09:46:16.5379922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5380362Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5380806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5381272Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5381749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5382232Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5382417Z 2025-12-04T09:46:16.5382530Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5382915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5383263Z res = mod(**inputs) 2025-12-04T09:46:16.5383684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5384113Z outputs = self.mobilebert( 2025-12-04T09:46:16.5384532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5384971Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5385407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5385850Z layer_outputs = layer_module( 2025-12-04T09:46:16.5386277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5386741Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5387206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5387709Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5388207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5388673Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5388833Z 2025-12-04T09:46:16.5388944Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5389329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5389662Z res = mod(**inputs) 2025-12-04T09:46:16.5390057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5390511Z outputs = self.mobilebert( 2025-12-04T09:46:16.5390962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5391427Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5391871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5392341Z layer_outputs = layer_module( 2025-12-04T09:46:16.5392777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5393268Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5393740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5394247Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5394747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5395262Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5395757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5396233Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5396394Z 2025-12-04T09:46:16.5396505Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5396893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5397239Z res = mod(**inputs) 2025-12-04T09:46:16.5397648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5398101Z outputs = self.mobilebert( 2025-12-04T09:46:16.5398533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5398984Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5399418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5399862Z layer_outputs = layer_module( 2025-12-04T09:46:16.5400306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5400836Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5401308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5401793Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5402286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5402744Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5402903Z 2025-12-04T09:46:16.5403015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5403403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5403753Z res = mod(**inputs) 2025-12-04T09:46:16.5404164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5404605Z outputs = self.mobilebert( 2025-12-04T09:46:16.5405040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5405478Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5405916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5406377Z layer_outputs = layer_module( 2025-12-04T09:46:16.5406843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5407312Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5407805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5408293Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5408749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5409196Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5409368Z 2025-12-04T09:46:16.5409471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5409836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5410186Z res = mod(**inputs) 2025-12-04T09:46:16.5410571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5410989Z outputs = self.mobilebert( 2025-12-04T09:46:16.5411396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5411808Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5412223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5412637Z layer_outputs = layer_module( 2025-12-04T09:46:16.5413048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5413482Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5413923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5414394Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5414866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5415315Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5415470Z 2025-12-04T09:46:16.5415580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5415963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5416307Z res = mod(**inputs) 2025-12-04T09:46:16.5416700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5417117Z outputs = self.mobilebert( 2025-12-04T09:46:16.5417523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5417935Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5418353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5418770Z layer_outputs = layer_module( 2025-12-04T09:46:16.5419177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5419622Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5420062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5420531Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5421012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5421505Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5421976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5422433Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5422586Z 2025-12-04T09:46:16.5422691Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5423055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5423387Z res = mod(**inputs) 2025-12-04T09:46:16.5423780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5424192Z outputs = self.mobilebert( 2025-12-04T09:46:16.5424601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5425041Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5425505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5425924Z layer_outputs = layer_module( 2025-12-04T09:46:16.5426334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5426802Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5427270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5427701Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5427842Z 2025-12-04T09:46:16.5427955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5428322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5428646Z res = mod(**inputs) 2025-12-04T09:46:16.5429035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5429455Z outputs = self.mobilebert( 2025-12-04T09:46:16.5429845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5430251Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5430667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5431082Z layer_outputs = layer_module( 2025-12-04T09:46:16.5431486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5431952Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5432419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5432898Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5433075Z 2025-12-04T09:46:16.5433188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5433569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5433914Z res = mod(**inputs) 2025-12-04T09:46:16.5434319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5434765Z outputs = self.mobilebert( 2025-12-04T09:46:16.5435182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5435602Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5436059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5436506Z layer_outputs = layer_module( 2025-12-04T09:46:16.5436963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5437499Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5438030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.5438495Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.5438660Z 2025-12-04T09:46:16.5438778Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5439158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5439524Z res = mod(**inputs) 2025-12-04T09:46:16.5439936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5440457Z outputs = self.mobilebert( 2025-12-04T09:46:16.5440902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5441357Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5441812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5442254Z layer_outputs = layer_module( 2025-12-04T09:46:16.5442682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5443210Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5443744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.5444231Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.5444735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5445200Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5445358Z 2025-12-04T09:46:16.5445476Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5445853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5446198Z res = mod(**inputs) 2025-12-04T09:46:16.5446608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5447241Z outputs = self.mobilebert( 2025-12-04T09:46:16.5447686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5448136Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5448583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5449020Z layer_outputs = layer_module( 2025-12-04T09:46:16.5449457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5449993Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5450530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5451045Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5451536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.5451958Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5452098Z 2025-12-04T09:46:16.5452234Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5452582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5452902Z res = mod(**inputs) 2025-12-04T09:46:16.5453286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5453688Z outputs = self.mobilebert( 2025-12-04T09:46:16.5454081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5454490Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5454911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5455312Z layer_outputs = layer_module( 2025-12-04T09:46:16.5455713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5456205Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5456697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5457148Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5457603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.5458059Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5458513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5458937Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5459090Z 2025-12-04T09:46:16.5459194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5459550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5459865Z res = mod(**inputs) 2025-12-04T09:46:16.5460250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5460660Z outputs = self.mobilebert( 2025-12-04T09:46:16.5461058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5461496Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5461914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5462340Z layer_outputs = layer_module( 2025-12-04T09:46:16.5462753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5463255Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5463762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5464225Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5464657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5465080Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5465249Z 2025-12-04T09:46:16.5465356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5465733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5466048Z res = mod(**inputs) 2025-12-04T09:46:16.5466452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5466862Z outputs = self.mobilebert( 2025-12-04T09:46:16.5467258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5467665Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5468070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5468483Z layer_outputs = layer_module( 2025-12-04T09:46:16.5468882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5469321Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5469740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5470153Z self_outputs = self.self( 2025-12-04T09:46:16.5470549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.5470970Z self.value(value_tensor) 2025-12-04T09:46:16.5471087Z 2025-12-04T09:46:16.5471199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5471563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5471880Z res = mod(**inputs) 2025-12-04T09:46:16.5472271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5472689Z outputs = self.mobilebert( 2025-12-04T09:46:16.5473086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5473509Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5473923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5474339Z layer_outputs = layer_module( 2025-12-04T09:46:16.5474741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5475244Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5475753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.5476208Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.5476655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5477084Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5477227Z 2025-12-04T09:46:16.5477342Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5477696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5478023Z res = mod(**inputs) 2025-12-04T09:46:16.5478412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5478835Z outputs = self.mobilebert( 2025-12-04T09:46:16.5479251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5479676Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5480110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5480599Z layer_outputs = layer_module( 2025-12-04T09:46:16.5481031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5481581Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5482142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5482636Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5483106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.5483580Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.5484026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5484470Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5484632Z 2025-12-04T09:46:16.5484741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5485109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5485440Z res = mod(**inputs) 2025-12-04T09:46:16.5485831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5486258Z outputs = self.mobilebert( 2025-12-04T09:46:16.5486675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5487088Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5487499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5487913Z layer_outputs = layer_module( 2025-12-04T09:46:16.5488321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5488743Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5489171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5489582Z self_outputs = self.self( 2025-12-04T09:46:16.5489981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.5490398Z self.query(query_tensor) 2025-12-04T09:46:16.5490521Z 2025-12-04T09:46:16.5490640Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5491000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5491319Z res = mod(**inputs) 2025-12-04T09:46:16.5491714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5492139Z outputs = self.mobilebert( 2025-12-04T09:46:16.5492550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5492968Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5493386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5493810Z layer_outputs = layer_module( 2025-12-04T09:46:16.5494261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5494710Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5495144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5495556Z self_outputs = self.self( 2025-12-04T09:46:16.5495961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.5496374Z self.key(key_tensor) 2025-12-04T09:46:16.5496491Z 2025-12-04T09:46:16.5496573Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5496791Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5497022Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5497382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5497711Z res = mod(**inputs) 2025-12-04T09:46:16.5498111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5498529Z outputs = self.mobilebert( 2025-12-04T09:46:16.5498929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5499345Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5499747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5500155Z layer_outputs = layer_module( 2025-12-04T09:46:16.5500557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5501063Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5501474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5501935Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5502393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.5502811Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5502955Z 2025-12-04T09:46:16.5503057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5503415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5503734Z res = mod(**inputs) 2025-12-04T09:46:16.5504115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5504526Z outputs = self.mobilebert( 2025-12-04T09:46:16.5504935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5505330Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5505729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5506138Z layer_outputs = layer_module( 2025-12-04T09:46:16.5506545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5506949Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5507356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5507801Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5508245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.5508699Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5509163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5509582Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5509724Z 2025-12-04T09:46:16.5509852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5510189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5510501Z res = mod(**inputs) 2025-12-04T09:46:16.5510867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5511257Z outputs = self.mobilebert( 2025-12-04T09:46:16.5511646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5512074Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5512476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5512874Z layer_outputs = layer_module( 2025-12-04T09:46:16.5513272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5513709Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5514149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5514600Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5515054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5515483Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5515627Z 2025-12-04T09:46:16.5515732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5516099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5516418Z res = mod(**inputs) 2025-12-04T09:46:16.5516800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5517202Z outputs = self.mobilebert( 2025-12-04T09:46:16.5517597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5518005Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5518409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5518806Z layer_outputs = layer_module( 2025-12-04T09:46:16.5519217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5519660Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5520096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5520702Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5521199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5521700Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5521878Z 2025-12-04T09:46:16.5521993Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5522388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5522719Z res = mod(**inputs) 2025-12-04T09:46:16.5523192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5523639Z outputs = self.mobilebert( 2025-12-04T09:46:16.5524085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5524535Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5524970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5525415Z layer_outputs = layer_module( 2025-12-04T09:46:16.5525849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5526322Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5526787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5527305Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5527801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5528256Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5528405Z 2025-12-04T09:46:16.5528515Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5528897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5529242Z res = mod(**inputs) 2025-12-04T09:46:16.5529647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5530089Z outputs = self.mobilebert( 2025-12-04T09:46:16.5530515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5530959Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5531387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5531826Z layer_outputs = layer_module( 2025-12-04T09:46:16.5532262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5532729Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5533191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5533659Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5534130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5534591Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5535061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5535527Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5535688Z 2025-12-04T09:46:16.5535806Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5536020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5536091Z res = mod(**inputs) 2025-12-04T09:46:16.5536407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5536485Z outputs = self.mobilebert( 2025-12-04T09:46:16.5536822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5536927Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5537225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5537313Z layer_outputs = layer_module( 2025-12-04T09:46:16.5537669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5537778Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5538085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5538214Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5538507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5538628Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5538633Z 2025-12-04T09:46:16.5538736Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5538947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5539013Z res = mod(**inputs) 2025-12-04T09:46:16.5539312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5539386Z outputs = self.mobilebert( 2025-12-04T09:46:16.5539676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5539761Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5540080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5540167Z layer_outputs = layer_module( 2025-12-04T09:46:16.5540476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5540579Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5540895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5541016Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5541332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5541460Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5541463Z 2025-12-04T09:46:16.5541574Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5541796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5541870Z res = mod(**inputs) 2025-12-04T09:46:16.5542182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5542265Z outputs = self.mobilebert( 2025-12-04T09:46:16.5542584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5542670Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5542979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5543061Z layer_outputs = layer_module( 2025-12-04T09:46:16.5543359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5543456Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5543769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5543934Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5544214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5544322Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5544326Z 2025-12-04T09:46:16.5544431Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5544633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5544709Z res = mod(**inputs) 2025-12-04T09:46:16.5544995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5545074Z outputs = self.mobilebert( 2025-12-04T09:46:16.5545356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5545452Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5545740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5545814Z layer_outputs = layer_module( 2025-12-04T09:46:16.5546097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5546200Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5546481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5546614Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5546896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5547021Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5547486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5547588Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5547592Z 2025-12-04T09:46:16.5547709Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5547908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5547974Z res = mod(**inputs) 2025-12-04T09:46:16.5548261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5548334Z outputs = self.mobilebert( 2025-12-04T09:46:16.5548627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5548704Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5549004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5549087Z layer_outputs = layer_module( 2025-12-04T09:46:16.5549397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5549490Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5549779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5549890Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5550216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5550305Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5550338Z 2025-12-04T09:46:16.5550442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5550649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5550714Z res = mod(**inputs) 2025-12-04T09:46:16.5551023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5551096Z outputs = self.mobilebert( 2025-12-04T09:46:16.5551378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5551461Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5551742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5551844Z layer_outputs = layer_module( 2025-12-04T09:46:16.5552140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5552235Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5552527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5552640Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5552925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5553047Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5553050Z 2025-12-04T09:46:16.5553154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5553365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5553433Z res = mod(**inputs) 2025-12-04T09:46:16.5553723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5553801Z outputs = self.mobilebert( 2025-12-04T09:46:16.5554079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5554161Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5554437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5554507Z layer_outputs = layer_module( 2025-12-04T09:46:16.5554790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5554880Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5555156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5555289Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5555565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5555655Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5555658Z 2025-12-04T09:46:16.5555760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5555952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5556026Z res = mod(**inputs) 2025-12-04T09:46:16.5556299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5556378Z outputs = self.mobilebert( 2025-12-04T09:46:16.5556676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5556773Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5557053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5557137Z layer_outputs = layer_module( 2025-12-04T09:46:16.5557415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5557511Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5557786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5557913Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5558190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5558330Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5558615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5558707Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5558710Z 2025-12-04T09:46:16.5558817Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5559013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5559079Z res = mod(**inputs) 2025-12-04T09:46:16.5559368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5559439Z outputs = self.mobilebert( 2025-12-04T09:46:16.5559723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5559806Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5560089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5560174Z layer_outputs = layer_module( 2025-12-04T09:46:16.5560538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5560677Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5560995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5561088Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5561093Z 2025-12-04T09:46:16.5561211Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5561440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5561508Z res = mod(**inputs) 2025-12-04T09:46:16.5561795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5561868Z outputs = self.mobilebert( 2025-12-04T09:46:16.5562156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5562232Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5562514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5562594Z layer_outputs = layer_module( 2025-12-04T09:46:16.5562876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5563030Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5563333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5563444Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5563447Z 2025-12-04T09:46:16.5563586Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5563782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5563848Z res = mod(**inputs) 2025-12-04T09:46:16.5564130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5564204Z outputs = self.mobilebert( 2025-12-04T09:46:16.5564488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5564576Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5564852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5564928Z layer_outputs = layer_module( 2025-12-04T09:46:16.5565204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5565361Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5565642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.5565735Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.5565738Z 2025-12-04T09:46:16.5565845Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5566038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5566106Z res = mod(**inputs) 2025-12-04T09:46:16.5566381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5566451Z outputs = self.mobilebert( 2025-12-04T09:46:16.5566728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5566800Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5567075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5567152Z layer_outputs = layer_module( 2025-12-04T09:46:16.5567423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5567580Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5567860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.5567984Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.5568264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5568355Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5568358Z 2025-12-04T09:46:16.5568456Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5568656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5568728Z res = mod(**inputs) 2025-12-04T09:46:16.5568997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5569106Z outputs = self.mobilebert( 2025-12-04T09:46:16.5569391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5569470Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5569791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5569870Z layer_outputs = layer_module( 2025-12-04T09:46:16.5570135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5570284Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5570560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5570684Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5570964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.5571055Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5571059Z 2025-12-04T09:46:16.5571158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5571355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5571418Z res = mod(**inputs) 2025-12-04T09:46:16.5571689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5571766Z outputs = self.mobilebert( 2025-12-04T09:46:16.5572036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5572118Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5572396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5572466Z layer_outputs = layer_module( 2025-12-04T09:46:16.5572749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5572899Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5573179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5573305Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5573567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.5573690Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5573958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5574044Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5574055Z 2025-12-04T09:46:16.5574152Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5574338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5574408Z res = mod(**inputs) 2025-12-04T09:46:16.5574670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5574736Z outputs = self.mobilebert( 2025-12-04T09:46:16.5575009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5575080Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5575369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5575456Z layer_outputs = layer_module( 2025-12-04T09:46:16.5575747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5575911Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5576181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5576289Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5576569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5576651Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5576683Z 2025-12-04T09:46:16.5576792Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5576982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5577045Z res = mod(**inputs) 2025-12-04T09:46:16.5577320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5577387Z outputs = self.mobilebert( 2025-12-04T09:46:16.5577659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5577730Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5577995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5578069Z layer_outputs = layer_module( 2025-12-04T09:46:16.5578338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5578425Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5578706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5578778Z self_outputs = self.self( 2025-12-04T09:46:16.5579056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.5579128Z self.value(value_tensor) 2025-12-04T09:46:16.5579131Z 2025-12-04T09:46:16.5579232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5579430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5579494Z res = mod(**inputs) 2025-12-04T09:46:16.5579777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5579848Z outputs = self.mobilebert( 2025-12-04T09:46:16.5580122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5580203Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5580480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5580547Z layer_outputs = layer_module( 2025-12-04T09:46:16.5580831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5580987Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5581288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.5581402Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.5581694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5581783Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5581787Z 2025-12-04T09:46:16.5581904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5582108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5582173Z res = mod(**inputs) 2025-12-04T09:46:16.5582455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5582532Z outputs = self.mobilebert( 2025-12-04T09:46:16.5582816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5582922Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5583203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5583274Z layer_outputs = layer_module( 2025-12-04T09:46:16.5583563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5583722Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5584004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5584130Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5584407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.5584500Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.5584780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5584874Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5584878Z 2025-12-04T09:46:16.5584991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5585191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5585263Z res = mod(**inputs) 2025-12-04T09:46:16.5585542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5585613Z outputs = self.mobilebert( 2025-12-04T09:46:16.5585900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5585977Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5586263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5586341Z layer_outputs = layer_module( 2025-12-04T09:46:16.5586622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5586730Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5587001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5587073Z self_outputs = self.self( 2025-12-04T09:46:16.5587352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.5587424Z self.query(query_tensor) 2025-12-04T09:46:16.5587427Z 2025-12-04T09:46:16.5587550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5587779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5587846Z res = mod(**inputs) 2025-12-04T09:46:16.5588132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5588220Z outputs = self.mobilebert( 2025-12-04T09:46:16.5588505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5588587Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5588870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5588948Z layer_outputs = layer_module( 2025-12-04T09:46:16.5589233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5589339Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5589630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5589713Z self_outputs = self.self( 2025-12-04T09:46:16.5589998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.5590065Z self.key(key_tensor) 2025-12-04T09:46:16.5590069Z 2025-12-04T09:46:16.5590149Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5590236Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5590338Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5590530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5590601Z res = mod(**inputs) 2025-12-04T09:46:16.5590878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5590957Z outputs = self.mobilebert( 2025-12-04T09:46:16.5591234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5591307Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5591595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5591667Z layer_outputs = layer_module( 2025-12-04T09:46:16.5591950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5592043Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5592326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5592462Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5592742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.5592825Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5592830Z 2025-12-04T09:46:16.5592941Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5593141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5593212Z res = mod(**inputs) 2025-12-04T09:46:16.5593502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5593572Z outputs = self.mobilebert( 2025-12-04T09:46:16.5593870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5593960Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5594240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5594312Z layer_outputs = layer_module( 2025-12-04T09:46:16.5594598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5594692Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5594972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5595089Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5595378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.5595523Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5595816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5595909Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5595913Z 2025-12-04T09:46:16.5596014Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5596219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5596283Z res = mod(**inputs) 2025-12-04T09:46:16.5596577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5596648Z outputs = self.mobilebert( 2025-12-04T09:46:16.5596934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5597018Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5597305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5597377Z layer_outputs = layer_module( 2025-12-04T09:46:16.5597669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5597762Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5598053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5598165Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5598457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5598547Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5598552Z 2025-12-04T09:46:16.5598652Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5598849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5598914Z res = mod(**inputs) 2025-12-04T09:46:16.5599224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5599307Z outputs = self.mobilebert( 2025-12-04T09:46:16.5599617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5599692Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5599986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5600057Z layer_outputs = layer_module( 2025-12-04T09:46:16.5600447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5600572Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5600865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5601013Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5601326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5601465Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5601469Z 2025-12-04T09:46:16.5601572Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5601767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5601844Z res = mod(**inputs) 2025-12-04T09:46:16.5602159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5602246Z outputs = self.mobilebert( 2025-12-04T09:46:16.5602544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5602625Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5602935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5603012Z layer_outputs = layer_module( 2025-12-04T09:46:16.5603378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5603490Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5603800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5603946Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5604241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5604333Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5604336Z 2025-12-04T09:46:16.5604453Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5604662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5604738Z res = mod(**inputs) 2025-12-04T09:46:16.5605042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5605117Z outputs = self.mobilebert( 2025-12-04T09:46:16.5605428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5605509Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5605808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5605891Z layer_outputs = layer_module( 2025-12-04T09:46:16.5606198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5606303Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5606601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5606733Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5607057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5607208Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5607514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5607612Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5607633Z 2025-12-04T09:46:16.5607744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5607965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5608036Z res = mod(**inputs) 2025-12-04T09:46:16.5608338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5608422Z outputs = self.mobilebert( 2025-12-04T09:46:16.5608720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5608818Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5609092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5609162Z layer_outputs = layer_module( 2025-12-04T09:46:16.5609449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5609537Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5609815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5609925Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5610200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5610290Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5610295Z 2025-12-04T09:46:16.5610394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5610591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5610655Z res = mod(**inputs) 2025-12-04T09:46:16.5610928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5611003Z outputs = self.mobilebert( 2025-12-04T09:46:16.5611277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5611346Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5611627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5611694Z layer_outputs = layer_module( 2025-12-04T09:46:16.5611977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5612070Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5612341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5612456Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5612732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5612848Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5612852Z 2025-12-04T09:46:16.5612953Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5613147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5613248Z res = mod(**inputs) 2025-12-04T09:46:16.5613538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5613608Z outputs = self.mobilebert( 2025-12-04T09:46:16.5613899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5613973Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5614264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5614332Z layer_outputs = layer_module( 2025-12-04T09:46:16.5614602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5614700Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5614969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5615113Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5615389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5615472Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5615476Z 2025-12-04T09:46:16.5615582Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5615776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5615840Z res = mod(**inputs) 2025-12-04T09:46:16.5616123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5616193Z outputs = self.mobilebert( 2025-12-04T09:46:16.5616484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5616559Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5616835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5616914Z layer_outputs = layer_module( 2025-12-04T09:46:16.5617192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5617289Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5617567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5617691Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5617978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5618098Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5618385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5618478Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5618481Z 2025-12-04T09:46:16.5618583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5618783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5618848Z res = mod(**inputs) 2025-12-04T09:46:16.5619129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5619206Z outputs = self.mobilebert( 2025-12-04T09:46:16.5619500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5619597Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5619875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5619945Z layer_outputs = layer_module( 2025-12-04T09:46:16.5620245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5620338Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5620622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5620732Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5621031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5621141Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5621146Z 2025-12-04T09:46:16.5621246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5621442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5621513Z res = mod(**inputs) 2025-12-04T09:46:16.5621792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5621869Z outputs = self.mobilebert( 2025-12-04T09:46:16.5622148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5622228Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5622508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5622581Z layer_outputs = layer_module( 2025-12-04T09:46:16.5622865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5622956Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5623230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5623344Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5623618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5623727Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5623737Z 2025-12-04T09:46:16.5623838Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5624034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5624106Z res = mod(**inputs) 2025-12-04T09:46:16.5624381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5624459Z outputs = self.mobilebert( 2025-12-04T09:46:16.5624730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5624801Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5625074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5625144Z layer_outputs = layer_module( 2025-12-04T09:46:16.5625419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5625518Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5625822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5625967Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5626246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5626328Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5626332Z 2025-12-04T09:46:16.5626438Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5626630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5626691Z res = mod(**inputs) 2025-12-04T09:46:16.5626969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5627035Z outputs = self.mobilebert( 2025-12-04T09:46:16.5627354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5627426Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5627689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5627766Z layer_outputs = layer_module( 2025-12-04T09:46:16.5628031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5628125Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5628388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5628504Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5628777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5628896Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5629160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5629253Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5629257Z 2025-12-04T09:46:16.5629352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5629544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5629606Z res = mod(**inputs) 2025-12-04T09:46:16.5629872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5629949Z outputs = self.mobilebert( 2025-12-04T09:46:16.5630220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5630300Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5630570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5630639Z layer_outputs = layer_module( 2025-12-04T09:46:16.5630915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5631032Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5631302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5631392Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5631396Z 2025-12-04T09:46:16.5631495Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5631712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5631791Z res = mod(**inputs) 2025-12-04T09:46:16.5632071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5632166Z outputs = self.mobilebert( 2025-12-04T09:46:16.5632451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5632532Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5632810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5632881Z layer_outputs = layer_module( 2025-12-04T09:46:16.5633170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5633323Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5633614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5633725Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5633729Z 2025-12-04T09:46:16.5633833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5634040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5634106Z res = mod(**inputs) 2025-12-04T09:46:16.5634387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5634467Z outputs = self.mobilebert( 2025-12-04T09:46:16.5634746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5634830Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5635114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5635186Z layer_outputs = layer_module( 2025-12-04T09:46:16.5635485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5635644Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5635924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.5636018Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.5636021Z 2025-12-04T09:46:16.5636122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5636340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5636406Z res = mod(**inputs) 2025-12-04T09:46:16.5636679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5636757Z outputs = self.mobilebert( 2025-12-04T09:46:16.5637029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5637110Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5637382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5637451Z layer_outputs = layer_module( 2025-12-04T09:46:16.5637732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5637906Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5638214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.5638351Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.5638639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5638738Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5638742Z 2025-12-04T09:46:16.5638841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5639042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5639106Z res = mod(**inputs) 2025-12-04T09:46:16.5639384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5639484Z outputs = self.mobilebert( 2025-12-04T09:46:16.5639768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5639840Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5640134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5640204Z layer_outputs = layer_module( 2025-12-04T09:46:16.5640577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5640743Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5641051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5641200Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5641510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.5641611Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5641616Z 2025-12-04T09:46:16.5641728Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5641944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5642025Z res = mod(**inputs) 2025-12-04T09:46:16.5642396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5642469Z outputs = self.mobilebert( 2025-12-04T09:46:16.5642758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5642833Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5643128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5643199Z layer_outputs = layer_module( 2025-12-04T09:46:16.5643469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5643648Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5643966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5644111Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5644418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.5644572Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5644890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5645026Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5645030Z 2025-12-04T09:46:16.5645151Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5645385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5645457Z res = mod(**inputs) 2025-12-04T09:46:16.5645778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5645855Z outputs = self.mobilebert( 2025-12-04T09:46:16.5646165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5646251Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5647355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5647449Z layer_outputs = layer_module( 2025-12-04T09:46:16.5647759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5647940Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5648257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5648380Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5648704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5648796Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5648802Z 2025-12-04T09:46:16.5648918Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5649148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5649218Z res = mod(**inputs) 2025-12-04T09:46:16.5649584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5649661Z outputs = self.mobilebert( 2025-12-04T09:46:16.5649925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5650004Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5650271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5650340Z layer_outputs = layer_module( 2025-12-04T09:46:16.5650613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5650700Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5650974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5651045Z self_outputs = self.self( 2025-12-04T09:46:16.5651310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.5651388Z self.value(value_tensor) 2025-12-04T09:46:16.5651391Z 2025-12-04T09:46:16.5651490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5651680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5651748Z res = mod(**inputs) 2025-12-04T09:46:16.5652058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5652159Z outputs = self.mobilebert( 2025-12-04T09:46:16.5652428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5652500Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5652801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5652871Z layer_outputs = layer_module( 2025-12-04T09:46:16.5653147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5653301Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5653568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.5653711Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.5653981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5654070Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5654074Z 2025-12-04T09:46:16.5654176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5654372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5654445Z res = mod(**inputs) 2025-12-04T09:46:16.5654717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5654785Z outputs = self.mobilebert( 2025-12-04T09:46:16.5655066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5655141Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5655420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5655490Z layer_outputs = layer_module( 2025-12-04T09:46:16.5655765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5655925Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5656198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5656314Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5656594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.5656678Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.5656952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5657040Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5657043Z 2025-12-04T09:46:16.5657141Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5657334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5657395Z res = mod(**inputs) 2025-12-04T09:46:16.5657666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5657733Z outputs = self.mobilebert( 2025-12-04T09:46:16.5657997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5658093Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5658377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5658452Z layer_outputs = layer_module( 2025-12-04T09:46:16.5658746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5658830Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5659106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5659175Z self_outputs = self.self( 2025-12-04T09:46:16.5659447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.5659522Z self.query(query_tensor) 2025-12-04T09:46:16.5659526Z 2025-12-04T09:46:16.5659624Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5659838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5659900Z res = mod(**inputs) 2025-12-04T09:46:16.5660166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5660242Z outputs = self.mobilebert( 2025-12-04T09:46:16.5660509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5660588Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5660854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5660921Z layer_outputs = layer_module( 2025-12-04T09:46:16.5661196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5661282Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5661551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5661628Z self_outputs = self.self( 2025-12-04T09:46:16.5661895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.5661965Z self.key(key_tensor) 2025-12-04T09:46:16.5661968Z 2025-12-04T09:46:16.5662047Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5662123Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5662229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5662417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5662480Z res = mod(**inputs) 2025-12-04T09:46:16.5662749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5662817Z outputs = self.mobilebert( 2025-12-04T09:46:16.5663086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5663158Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5663423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5663497Z layer_outputs = layer_module( 2025-12-04T09:46:16.5663760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5663845Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5664124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5664261Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5664537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.5664616Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5664633Z 2025-12-04T09:46:16.5664740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5664928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5664991Z res = mod(**inputs) 2025-12-04T09:46:16.5665261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5665329Z outputs = self.mobilebert( 2025-12-04T09:46:16.5665595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5665690Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5665958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5666033Z layer_outputs = layer_module( 2025-12-04T09:46:16.5666298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5666379Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5666653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5666771Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5667053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.5667181Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5667455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5667552Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5667555Z 2025-12-04T09:46:16.5667655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5667849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5667920Z res = mod(**inputs) 2025-12-04T09:46:16.5668194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5668269Z outputs = self.mobilebert( 2025-12-04T09:46:16.5668543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5668619Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5668901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5668970Z layer_outputs = layer_module( 2025-12-04T09:46:16.5669253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5669348Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5669621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5669739Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5670010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5670092Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5670121Z 2025-12-04T09:46:16.5670237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5670430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5670501Z res = mod(**inputs) 2025-12-04T09:46:16.5670788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5670860Z outputs = self.mobilebert( 2025-12-04T09:46:16.5671140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5671213Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5671493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5671562Z layer_outputs = layer_module( 2025-12-04T09:46:16.5671861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5671963Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5672228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5672335Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5672607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5672715Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5672718Z 2025-12-04T09:46:16.5672821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5673008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5673072Z res = mod(**inputs) 2025-12-04T09:46:16.5673346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5673414Z outputs = self.mobilebert( 2025-12-04T09:46:16.5673686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5673757Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5674022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5674098Z layer_outputs = layer_module( 2025-12-04T09:46:16.5674361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5674457Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5674723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5674847Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5675117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5675198Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5675201Z 2025-12-04T09:46:16.5675299Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5675494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5675556Z res = mod(**inputs) 2025-12-04T09:46:16.5675827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5675895Z outputs = self.mobilebert( 2025-12-04T09:46:16.5676180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5676279Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5676548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5676623Z layer_outputs = layer_module( 2025-12-04T09:46:16.5676908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5676998Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5677272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5677391Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5677658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5677798Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5678073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5678168Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5678173Z 2025-12-04T09:46:16.5678275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5678470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5678539Z res = mod(**inputs) 2025-12-04T09:46:16.5678811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5678885Z outputs = self.mobilebert( 2025-12-04T09:46:16.5679155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5679229Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5679508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5679577Z layer_outputs = layer_module( 2025-12-04T09:46:16.5679851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5679950Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5680221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5680397Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5680683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5680769Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5680774Z 2025-12-04T09:46:16.5680884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5681080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5681150Z res = mod(**inputs) 2025-12-04T09:46:16.5681427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5681503Z outputs = self.mobilebert( 2025-12-04T09:46:16.5681812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5681890Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5682210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5682286Z layer_outputs = layer_module( 2025-12-04T09:46:16.5682612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5682752Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5683079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5683207Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5683515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5683629Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5683633Z 2025-12-04T09:46:16.5683748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5683947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5684036Z res = mod(**inputs) 2025-12-04T09:46:16.5684325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5684398Z outputs = self.mobilebert( 2025-12-04T09:46:16.5684691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5684777Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5685051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5685129Z layer_outputs = layer_module( 2025-12-04T09:46:16.5685406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5685497Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5685785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5685908Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5686192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5686275Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5686278Z 2025-12-04T09:46:16.5686377Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5686575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5686638Z res = mod(**inputs) 2025-12-04T09:46:16.5686909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5686977Z outputs = self.mobilebert( 2025-12-04T09:46:16.5687248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5687329Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5687613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5687692Z layer_outputs = layer_module( 2025-12-04T09:46:16.5687976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5688068Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5688351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5688472Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5688765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5688911Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5689188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5689304Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5689308Z 2025-12-04T09:46:16.5689411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5689607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5689681Z res = mod(**inputs) 2025-12-04T09:46:16.5689961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5690040Z outputs = self.mobilebert( 2025-12-04T09:46:16.5690320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5690413Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5690695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5690764Z layer_outputs = layer_module( 2025-12-04T09:46:16.5691042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5691144Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5691420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5691536Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5691813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5691897Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5691902Z 2025-12-04T09:46:16.5692009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5692202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5692273Z res = mod(**inputs) 2025-12-04T09:46:16.5692550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5692619Z outputs = self.mobilebert( 2025-12-04T09:46:16.5692902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5692977Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5693253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5693334Z layer_outputs = layer_module( 2025-12-04T09:46:16.5693610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5693708Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5693985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5694095Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5694381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5694490Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5694494Z 2025-12-04T09:46:16.5694599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5694792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5694875Z res = mod(**inputs) 2025-12-04T09:46:16.5695178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5695247Z outputs = self.mobilebert( 2025-12-04T09:46:16.5695534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5695616Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5695891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5695969Z layer_outputs = layer_module( 2025-12-04T09:46:16.5696248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5696342Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5696641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5696764Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5697045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5697127Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5697131Z 2025-12-04T09:46:16.5697230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5697427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5697490Z res = mod(**inputs) 2025-12-04T09:46:16.5697768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5697835Z outputs = self.mobilebert( 2025-12-04T09:46:16.5698112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5698193Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5698467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5698538Z layer_outputs = layer_module( 2025-12-04T09:46:16.5698821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5698911Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5699189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5699309Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5699591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5699720Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5699985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5700080Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5700083Z 2025-12-04T09:46:16.5700181Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5700366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5700438Z res = mod(**inputs) 2025-12-04T09:46:16.5700701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5700769Z outputs = self.mobilebert( 2025-12-04T09:46:16.5701062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5701151Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5701426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5701510Z layer_outputs = layer_module( 2025-12-04T09:46:16.5701778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5701903Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5702175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5702263Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5702266Z 2025-12-04T09:46:16.5702365Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5702577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5702650Z res = mod(**inputs) 2025-12-04T09:46:16.5702924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5702993Z outputs = self.mobilebert( 2025-12-04T09:46:16.5703280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5703351Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5703630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5703700Z layer_outputs = layer_module( 2025-12-04T09:46:16.5703976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5704105Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5704385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5704502Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5704505Z 2025-12-04T09:46:16.5704607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5704803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5704873Z res = mod(**inputs) 2025-12-04T09:46:16.5705148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5705226Z outputs = self.mobilebert( 2025-12-04T09:46:16.5705501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5705575Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5705857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5705925Z layer_outputs = layer_module( 2025-12-04T09:46:16.5706202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5706365Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5706639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.5706740Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.5706744Z 2025-12-04T09:46:16.5706844Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5707066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5707156Z res = mod(**inputs) 2025-12-04T09:46:16.5707434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5707509Z outputs = self.mobilebert( 2025-12-04T09:46:16.5707801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5707875Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5708156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5708225Z layer_outputs = layer_module( 2025-12-04T09:46:16.5708495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5708659Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5708951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.5709081Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.5709370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5709460Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5709464Z 2025-12-04T09:46:16.5709572Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5709766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5709833Z res = mod(**inputs) 2025-12-04T09:46:16.5710100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5710171Z outputs = self.mobilebert( 2025-12-04T09:46:16.5710445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5710516Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5710784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5710857Z layer_outputs = layer_module( 2025-12-04T09:46:16.5711123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5711279Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5711546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5711664Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5711938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.5712021Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5712024Z 2025-12-04T09:46:16.5712126Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5712314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5712375Z res = mod(**inputs) 2025-12-04T09:46:16.5712650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5712719Z outputs = self.mobilebert( 2025-12-04T09:46:16.5712992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5713064Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5713349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5713443Z layer_outputs = layer_module( 2025-12-04T09:46:16.5713726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5713877Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5714151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5714271Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5714543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.5714659Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5714943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5715041Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5715045Z 2025-12-04T09:46:16.5715144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5715341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5715404Z res = mod(**inputs) 2025-12-04T09:46:16.5715671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5715748Z outputs = self.mobilebert( 2025-12-04T09:46:16.5716015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5716092Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5716360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5716429Z layer_outputs = layer_module( 2025-12-04T09:46:16.5716701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5716856Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5717123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5717236Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5717502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5717587Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5717592Z 2025-12-04T09:46:16.5717691Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5717880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5717953Z res = mod(**inputs) 2025-12-04T09:46:16.5718219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5718294Z outputs = self.mobilebert( 2025-12-04T09:46:16.5718558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5718630Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5718910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5718980Z layer_outputs = layer_module( 2025-12-04T09:46:16.5719273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5719384Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5719657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5719754Z self_outputs = self.self( 2025-12-04T09:46:16.5720030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.5720101Z self.value(value_tensor) 2025-12-04T09:46:16.5720105Z 2025-12-04T09:46:16.5720212Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5720618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5720700Z res = mod(**inputs) 2025-12-04T09:46:16.5720974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5721066Z outputs = self.mobilebert( 2025-12-04T09:46:16.5721347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5721428Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5721745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5721836Z layer_outputs = layer_module( 2025-12-04T09:46:16.5722143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5722326Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5722641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.5722767Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.5723081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5723169Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5723173Z 2025-12-04T09:46:16.5723295Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5723505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5723569Z res = mod(**inputs) 2025-12-04T09:46:16.5723847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5723916Z outputs = self.mobilebert( 2025-12-04T09:46:16.5724186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5724269Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5724542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5724618Z layer_outputs = layer_module( 2025-12-04T09:46:16.5724893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5725048Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5725332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5725438Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5725719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.5725825Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.5726115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5726214Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5726218Z 2025-12-04T09:46:16.5726365Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5726569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5726633Z res = mod(**inputs) 2025-12-04T09:46:16.5726906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5726981Z outputs = self.mobilebert( 2025-12-04T09:46:16.5727256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5727330Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5727635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5727704Z layer_outputs = layer_module( 2025-12-04T09:46:16.5727982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5728067Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5728344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5728419Z self_outputs = self.self( 2025-12-04T09:46:16.5728693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.5728770Z self.query(query_tensor) 2025-12-04T09:46:16.5728773Z 2025-12-04T09:46:16.5728873Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5729068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5729138Z res = mod(**inputs) 2025-12-04T09:46:16.5729408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5729478Z outputs = self.mobilebert( 2025-12-04T09:46:16.5729762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5729834Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5730118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5730187Z layer_outputs = layer_module( 2025-12-04T09:46:16.5730464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5730558Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5730830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5730906Z self_outputs = self.self( 2025-12-04T09:46:16.5731183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.5731250Z self.key(key_tensor) 2025-12-04T09:46:16.5731253Z 2025-12-04T09:46:16.5731344Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5731420Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5731522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5731721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5731785Z res = mod(**inputs) 2025-12-04T09:46:16.5732083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5732169Z outputs = self.mobilebert( 2025-12-04T09:46:16.5732445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5732541Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5732817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5732887Z layer_outputs = layer_module( 2025-12-04T09:46:16.5733171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5733255Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5733586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5733725Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5733999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.5734088Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5734093Z 2025-12-04T09:46:16.5734193Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5734397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5734462Z res = mod(**inputs) 2025-12-04T09:46:16.5734735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5734812Z outputs = self.mobilebert( 2025-12-04T09:46:16.5735087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5735162Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5735449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5735517Z layer_outputs = layer_module( 2025-12-04T09:46:16.5735799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5735882Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5736157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5736285Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5736558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.5736691Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5736967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5737057Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5737061Z 2025-12-04T09:46:16.5737171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5737362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5737432Z res = mod(**inputs) 2025-12-04T09:46:16.5737707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5737776Z outputs = self.mobilebert( 2025-12-04T09:46:16.5738061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5738151Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5738445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5738522Z layer_outputs = layer_module( 2025-12-04T09:46:16.5738811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5738914Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5739190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5739302Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5739585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5739667Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5739688Z 2025-12-04T09:46:16.5739796Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5739990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5740054Z res = mod(**inputs) 2025-12-04T09:46:16.5740336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5740406Z outputs = self.mobilebert( 2025-12-04T09:46:16.5740732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5740811Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5741095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5741173Z layer_outputs = layer_module( 2025-12-04T09:46:16.5741458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5741556Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5741888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5742001Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5742290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5742403Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5742407Z 2025-12-04T09:46:16.5742510Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5742717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5742782Z res = mod(**inputs) 2025-12-04T09:46:16.5743066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5743145Z outputs = self.mobilebert( 2025-12-04T09:46:16.5743427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5743506Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5743789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5743858Z layer_outputs = layer_module( 2025-12-04T09:46:16.5744147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5744242Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5744544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5744689Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5744971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5745079Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5745083Z 2025-12-04T09:46:16.5745187Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5745395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5745460Z res = mod(**inputs) 2025-12-04T09:46:16.5745742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5745819Z outputs = self.mobilebert( 2025-12-04T09:46:16.5746103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5746199Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5746498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5746569Z layer_outputs = layer_module( 2025-12-04T09:46:16.5746869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5746963Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5747404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5747543Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5747829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5747962Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5748245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5748337Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5748341Z 2025-12-04T09:46:16.5748454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5748651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5748718Z res = mod(**inputs) 2025-12-04T09:46:16.5749008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5749079Z outputs = self.mobilebert( 2025-12-04T09:46:16.5749367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5749443Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5749729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5749809Z layer_outputs = layer_module( 2025-12-04T09:46:16.5750091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5750195Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5750476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5750590Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5750877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5750997Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5751023Z 2025-12-04T09:46:16.5751134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5751334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5751400Z res = mod(**inputs) 2025-12-04T09:46:16.5751728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5751803Z outputs = self.mobilebert( 2025-12-04T09:46:16.5752087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5752170Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5752454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5752542Z layer_outputs = layer_module( 2025-12-04T09:46:16.5752853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5752948Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5753234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5753343Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5753628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5753742Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5753746Z 2025-12-04T09:46:16.5753849Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5754055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5754124Z res = mod(**inputs) 2025-12-04T09:46:16.5754405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5754487Z outputs = self.mobilebert( 2025-12-04T09:46:16.5754772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5754855Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5755138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5755210Z layer_outputs = layer_module( 2025-12-04T09:46:16.5755495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5755587Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5755873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5756001Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5756279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5756372Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5756376Z 2025-12-04T09:46:16.5756479Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5756678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5756752Z res = mod(**inputs) 2025-12-04T09:46:16.5757034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5757112Z outputs = self.mobilebert( 2025-12-04T09:46:16.5757410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5757505Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5757801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5757886Z layer_outputs = layer_module( 2025-12-04T09:46:16.5758187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5758285Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5758582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5758720Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5759018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5759173Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5759471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5759571Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5759575Z 2025-12-04T09:46:16.5759689Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5759904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5759973Z res = mod(**inputs) 2025-12-04T09:46:16.5760283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5760407Z outputs = self.mobilebert( 2025-12-04T09:46:16.5760728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5760812Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5761119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5761204Z layer_outputs = layer_module( 2025-12-04T09:46:16.5761518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5761623Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5761922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5762039Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5762343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5762435Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5762440Z 2025-12-04T09:46:16.5762548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5762767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5762831Z res = mod(**inputs) 2025-12-04T09:46:16.5763113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5763182Z outputs = self.mobilebert( 2025-12-04T09:46:16.5763455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5763537Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5763811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5763912Z layer_outputs = layer_module( 2025-12-04T09:46:16.5764204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5764296Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5764594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5764704Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5764981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5765098Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5765101Z 2025-12-04T09:46:16.5765201Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5765404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5765487Z res = mod(**inputs) 2025-12-04T09:46:16.5765772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5765850Z outputs = self.mobilebert( 2025-12-04T09:46:16.5766134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5766214Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5766499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5766569Z layer_outputs = layer_module( 2025-12-04T09:46:16.5766860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5766950Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5767240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5767374Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5767660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5767748Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5767751Z 2025-12-04T09:46:16.5767852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5768049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5768122Z res = mod(**inputs) 2025-12-04T09:46:16.5768405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5768483Z outputs = self.mobilebert( 2025-12-04T09:46:16.5768772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5768846Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5769144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5769215Z layer_outputs = layer_module( 2025-12-04T09:46:16.5769507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5769597Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5769882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5770011Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5770317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5770456Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5770740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5770845Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5770849Z 2025-12-04T09:46:16.5770958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5771154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5771218Z res = mod(**inputs) 2025-12-04T09:46:16.5771501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5771571Z outputs = self.mobilebert( 2025-12-04T09:46:16.5771856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5771947Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5772222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5772299Z layer_outputs = layer_module( 2025-12-04T09:46:16.5772582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5772698Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5772971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5773051Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5773054Z 2025-12-04T09:46:16.5773155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5773343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5773407Z res = mod(**inputs) 2025-12-04T09:46:16.5773676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5773744Z outputs = self.mobilebert( 2025-12-04T09:46:16.5774015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5774085Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5774352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5774429Z layer_outputs = layer_module( 2025-12-04T09:46:16.5774700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5774819Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5775105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5775211Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5775215Z 2025-12-04T09:46:16.5775322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5775514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5775577Z res = mod(**inputs) 2025-12-04T09:46:16.5775864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5775935Z outputs = self.mobilebert( 2025-12-04T09:46:16.5776228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5776320Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5776616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5776691Z layer_outputs = layer_module( 2025-12-04T09:46:16.5776972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5777133Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5777408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.5777503Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.5777506Z 2025-12-04T09:46:16.5777615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5777814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5777894Z res = mod(**inputs) 2025-12-04T09:46:16.5778178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5778246Z outputs = self.mobilebert( 2025-12-04T09:46:16.5778568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5778641Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5778925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5779002Z layer_outputs = layer_module( 2025-12-04T09:46:16.5779286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5779451Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5779738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.5779861Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.5780153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5780243Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5780247Z 2025-12-04T09:46:16.5780355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5780566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5780628Z res = mod(**inputs) 2025-12-04T09:46:16.5780912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5780983Z outputs = self.mobilebert( 2025-12-04T09:46:16.5781261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5781338Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5781645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5781722Z layer_outputs = layer_module( 2025-12-04T09:46:16.5782005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5782159Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5782454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5782588Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5782868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.5782969Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5782972Z 2025-12-04T09:46:16.5783069Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5783278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5783342Z res = mod(**inputs) 2025-12-04T09:46:16.5783614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5783692Z outputs = self.mobilebert( 2025-12-04T09:46:16.5783970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5784049Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5784345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5784416Z layer_outputs = layer_module( 2025-12-04T09:46:16.5784700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5784856Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5785152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5785283Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5785550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.5785674Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5785949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5786044Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5786048Z 2025-12-04T09:46:16.5786147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5786339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5786410Z res = mod(**inputs) 2025-12-04T09:46:16.5786688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5786760Z outputs = self.mobilebert( 2025-12-04T09:46:16.5787044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5787116Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5787399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5787469Z layer_outputs = layer_module( 2025-12-04T09:46:16.5787744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5787911Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5788191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5788306Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5788582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5788664Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5788669Z 2025-12-04T09:46:16.5788792Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5789005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5789069Z res = mod(**inputs) 2025-12-04T09:46:16.5789368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5789440Z outputs = self.mobilebert( 2025-12-04T09:46:16.5789721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5789795Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5790071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5790147Z layer_outputs = layer_module( 2025-12-04T09:46:16.5790427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5790543Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5790817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5790891Z self_outputs = self.self( 2025-12-04T09:46:16.5791179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.5791253Z self.value(value_tensor) 2025-12-04T09:46:16.5791256Z 2025-12-04T09:46:16.5791358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5791559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5791623Z res = mod(**inputs) 2025-12-04T09:46:16.5791910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5791983Z outputs = self.mobilebert( 2025-12-04T09:46:16.5792258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5792339Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5792629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5792704Z layer_outputs = layer_module( 2025-12-04T09:46:16.5792980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5793134Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5793413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.5793523Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.5793795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5793885Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5793889Z 2025-12-04T09:46:16.5793992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5794195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5794259Z res = mod(**inputs) 2025-12-04T09:46:16.5794542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5794619Z outputs = self.mobilebert( 2025-12-04T09:46:16.5794890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5794984Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5795275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5795345Z layer_outputs = layer_module( 2025-12-04T09:46:16.5795644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5795807Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5796100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5796209Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5796496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.5796608Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.5796891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5796984Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5796994Z 2025-12-04T09:46:16.5797100Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5797298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5797370Z res = mod(**inputs) 2025-12-04T09:46:16.5797648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5797719Z outputs = self.mobilebert( 2025-12-04T09:46:16.5798005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5798081Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5798378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5798456Z layer_outputs = layer_module( 2025-12-04T09:46:16.5798751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5798851Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5799146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5799222Z self_outputs = self.self( 2025-12-04T09:46:16.5799527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.5799605Z self.query(query_tensor) 2025-12-04T09:46:16.5799608Z 2025-12-04T09:46:16.5799725Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5799937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5800007Z res = mod(**inputs) 2025-12-04T09:46:16.5800310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5800478Z outputs = self.mobilebert( 2025-12-04T09:46:16.5800798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5800879Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5801181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5801267Z layer_outputs = layer_module( 2025-12-04T09:46:16.5801603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5801712Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5802013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5802085Z self_outputs = self.self( 2025-12-04T09:46:16.5802397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.5802469Z self.key(key_tensor) 2025-12-04T09:46:16.5802473Z 2025-12-04T09:46:16.5802557Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5802645Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5802746Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5802939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5803009Z res = mod(**inputs) 2025-12-04T09:46:16.5803283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5803380Z outputs = self.mobilebert( 2025-12-04T09:46:16.5803661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5803733Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5804020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5804090Z layer_outputs = layer_module( 2025-12-04T09:46:16.5804378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5804464Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5804749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5804881Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5805163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.5805244Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5805256Z 2025-12-04T09:46:16.5805356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5805552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5805624Z res = mod(**inputs) 2025-12-04T09:46:16.5805904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5805973Z outputs = self.mobilebert( 2025-12-04T09:46:16.5806267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5806342Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5806629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5806697Z layer_outputs = layer_module( 2025-12-04T09:46:16.5806980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5807069Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5807351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5807469Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5807761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.5807905Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5808203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5808297Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5808301Z 2025-12-04T09:46:16.5808417Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5808620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5808683Z res = mod(**inputs) 2025-12-04T09:46:16.5808961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5809031Z outputs = self.mobilebert( 2025-12-04T09:46:16.5809304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5809403Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5809684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5809759Z layer_outputs = layer_module( 2025-12-04T09:46:16.5810040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5810133Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5810421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5810530Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5810819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5810908Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5810914Z 2025-12-04T09:46:16.5811015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5811225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5811289Z res = mod(**inputs) 2025-12-04T09:46:16.5811580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5811659Z outputs = self.mobilebert( 2025-12-04T09:46:16.5811957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5812036Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5812316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5812386Z layer_outputs = layer_module( 2025-12-04T09:46:16.5812678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5812775Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5813055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5813174Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5813452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5813569Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5813573Z 2025-12-04T09:46:16.5813673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5813867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5813940Z res = mod(**inputs) 2025-12-04T09:46:16.5814241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5814337Z outputs = self.mobilebert( 2025-12-04T09:46:16.5814613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5814701Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5814985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5815055Z layer_outputs = layer_module( 2025-12-04T09:46:16.5815329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5815429Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5815706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5815860Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5816131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5816216Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5816219Z 2025-12-04T09:46:16.5816324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5816519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5816592Z res = mod(**inputs) 2025-12-04T09:46:16.5816868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5816941Z outputs = self.mobilebert( 2025-12-04T09:46:16.5817225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5817302Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5817589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5817663Z layer_outputs = layer_module( 2025-12-04T09:46:16.5817941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5818050Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5818326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5818452Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5818738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5818864Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5819149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5819242Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5819245Z 2025-12-04T09:46:16.5819352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5819554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5819622Z res = mod(**inputs) 2025-12-04T09:46:16.5819914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5819991Z outputs = self.mobilebert( 2025-12-04T09:46:16.5820277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5820381Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5820683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5820754Z layer_outputs = layer_module( 2025-12-04T09:46:16.5821063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5821160Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5821448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5821571Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5821849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5821945Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5821975Z 2025-12-04T09:46:16.5822077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5822283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5822348Z res = mod(**inputs) 2025-12-04T09:46:16.5822676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5822756Z outputs = self.mobilebert( 2025-12-04T09:46:16.5823039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5823112Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5823411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5823479Z layer_outputs = layer_module( 2025-12-04T09:46:16.5823763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5823856Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5824138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5824258Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5824540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5824668Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5824672Z 2025-12-04T09:46:16.5824770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5824967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5825038Z res = mod(**inputs) 2025-12-04T09:46:16.5825316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5825393Z outputs = self.mobilebert( 2025-12-04T09:46:16.5825671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5825743Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5826027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5826096Z layer_outputs = layer_module( 2025-12-04T09:46:16.5826373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5826472Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5826765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5826943Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5827224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5827325Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5827329Z 2025-12-04T09:46:16.5827442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5827641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5827723Z res = mod(**inputs) 2025-12-04T09:46:16.5827998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5828067Z outputs = self.mobilebert( 2025-12-04T09:46:16.5828352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5828444Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5828727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5828806Z layer_outputs = layer_module( 2025-12-04T09:46:16.5829090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5829190Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5829470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5829595Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5829888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5830014Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5830302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5830396Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5830400Z 2025-12-04T09:46:16.5830506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5830712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5830778Z res = mod(**inputs) 2025-12-04T09:46:16.5831067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5831139Z outputs = self.mobilebert( 2025-12-04T09:46:16.5831442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5831532Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5831831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5831906Z layer_outputs = layer_module( 2025-12-04T09:46:16.5832224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5832324Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5832638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5832756Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5833071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5833190Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5833210Z 2025-12-04T09:46:16.5833320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5833537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5833609Z res = mod(**inputs) 2025-12-04T09:46:16.5833933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5834019Z outputs = self.mobilebert( 2025-12-04T09:46:16.5834330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5834408Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5834722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5834798Z layer_outputs = layer_module( 2025-12-04T09:46:16.5835123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5835224Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5835519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5835645Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5835951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5836075Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5836078Z 2025-12-04T09:46:16.5836187Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5836396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5836476Z res = mod(**inputs) 2025-12-04T09:46:16.5836772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5836847Z outputs = self.mobilebert( 2025-12-04T09:46:16.5837163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5837240Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5837545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5837618Z layer_outputs = layer_module( 2025-12-04T09:46:16.5837924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5838035Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5838338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5838487Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5838791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5838884Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5838888Z 2025-12-04T09:46:16.5839008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5839223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5839301Z res = mod(**inputs) 2025-12-04T09:46:16.5839606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5839684Z outputs = self.mobilebert( 2025-12-04T09:46:16.5840017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5840116Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5840500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5840619Z layer_outputs = layer_module( 2025-12-04T09:46:16.5840932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5841045Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5841355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5841502Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5841816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5841970Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5842275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5842374Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5842377Z 2025-12-04T09:46:16.5842486Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5842701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5842770Z res = mod(**inputs) 2025-12-04T09:46:16.5843065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5843153Z outputs = self.mobilebert( 2025-12-04T09:46:16.5843450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5843539Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5843836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5843911Z layer_outputs = layer_module( 2025-12-04T09:46:16.5844217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5844350Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5844652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5844742Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5844746Z 2025-12-04T09:46:16.5844854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5845073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5845145Z res = mod(**inputs) 2025-12-04T09:46:16.5845442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5845524Z outputs = self.mobilebert( 2025-12-04T09:46:16.5845822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5845907Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5846204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5846279Z layer_outputs = layer_module( 2025-12-04T09:46:16.5846581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5846728Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5847169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5847299Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5847304Z 2025-12-04T09:46:16.5847461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5847687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5847757Z res = mod(**inputs) 2025-12-04T09:46:16.5848057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5848141Z outputs = self.mobilebert( 2025-12-04T09:46:16.5848442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5848555Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5848852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5848928Z layer_outputs = layer_module( 2025-12-04T09:46:16.5849235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5849402Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5849708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.5849811Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.5849814Z 2025-12-04T09:46:16.5849925Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5850152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5850222Z res = mod(**inputs) 2025-12-04T09:46:16.5850512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5850585Z outputs = self.mobilebert( 2025-12-04T09:46:16.5850869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5850952Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5851230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5851301Z layer_outputs = layer_module( 2025-12-04T09:46:16.5851590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5851748Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5852037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.5852162Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.5852444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5852545Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5852548Z 2025-12-04T09:46:16.5852651Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5852853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5852919Z res = mod(**inputs) 2025-12-04T09:46:16.5853198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5853307Z outputs = self.mobilebert( 2025-12-04T09:46:16.5853610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5853683Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5853988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5854062Z layer_outputs = layer_module( 2025-12-04T09:46:16.5854353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5854511Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5854790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5854927Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5855231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.5855324Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5855328Z 2025-12-04T09:46:16.5855430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5855630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5855704Z res = mod(**inputs) 2025-12-04T09:46:16.5855986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5856067Z outputs = self.mobilebert( 2025-12-04T09:46:16.5856353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5856427Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5856722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5856796Z layer_outputs = layer_module( 2025-12-04T09:46:16.5857079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5857244Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5857529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5857659Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5857942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.5858066Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5858361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5858455Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5858458Z 2025-12-04T09:46:16.5858566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5858767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5858833Z res = mod(**inputs) 2025-12-04T09:46:16.5859124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5859195Z outputs = self.mobilebert( 2025-12-04T09:46:16.5859488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5859562Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5859864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5859958Z layer_outputs = layer_module( 2025-12-04T09:46:16.5860240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5860414Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5860707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5860818Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5861103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5861185Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5861189Z 2025-12-04T09:46:16.5861311Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5861516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5861580Z res = mod(**inputs) 2025-12-04T09:46:16.5861871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5861944Z outputs = self.mobilebert( 2025-12-04T09:46:16.5862237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5862315Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5862598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5862668Z layer_outputs = layer_module( 2025-12-04T09:46:16.5862957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5863044Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5863319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5863388Z self_outputs = self.self( 2025-12-04T09:46:16.5863664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.5863743Z self.value(value_tensor) 2025-12-04T09:46:16.5863746Z 2025-12-04T09:46:16.5863847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5864045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5864110Z res = mod(**inputs) 2025-12-04T09:46:16.5864392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5864473Z outputs = self.mobilebert( 2025-12-04T09:46:16.5864755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5864828Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5865119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5865192Z layer_outputs = layer_module( 2025-12-04T09:46:16.5865482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5865641Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5865922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.5866066Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.5866347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5866435Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5866439Z 2025-12-04T09:46:16.5866554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5866742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5866814Z res = mod(**inputs) 2025-12-04T09:46:16.5867077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5867148Z outputs = self.mobilebert( 2025-12-04T09:46:16.5867428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5867518Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5867814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5867884Z layer_outputs = layer_module( 2025-12-04T09:46:16.5868163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5868326Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5868605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5868719Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5869003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.5869092Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.5869381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5869474Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5869478Z 2025-12-04T09:46:16.5869585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5869785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5869852Z res = mod(**inputs) 2025-12-04T09:46:16.5870155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5870227Z outputs = self.mobilebert( 2025-12-04T09:46:16.5870505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5870586Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5870861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5870940Z layer_outputs = layer_module( 2025-12-04T09:46:16.5871219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5871305Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5871589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5871661Z self_outputs = self.self( 2025-12-04T09:46:16.5871945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.5872017Z self.query(query_tensor) 2025-12-04T09:46:16.5872020Z 2025-12-04T09:46:16.5872137Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5872333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5872425Z res = mod(**inputs) 2025-12-04T09:46:16.5872694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5872783Z outputs = self.mobilebert( 2025-12-04T09:46:16.5873052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5873131Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5873410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5873479Z layer_outputs = layer_module( 2025-12-04T09:46:16.5873767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5873872Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5874158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5874231Z self_outputs = self.self( 2025-12-04T09:46:16.5874514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.5874590Z self.key(key_tensor) 2025-12-04T09:46:16.5874594Z 2025-12-04T09:46:16.5874678Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5874759Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5874879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5875077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5875148Z res = mod(**inputs) 2025-12-04T09:46:16.5875425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5875494Z outputs = self.mobilebert( 2025-12-04T09:46:16.5875772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5875845Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5876120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5876196Z layer_outputs = layer_module( 2025-12-04T09:46:16.5876473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5876561Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5876842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5876970Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5877256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.5877346Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5877349Z 2025-12-04T09:46:16.5877463Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5877674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5877742Z res = mod(**inputs) 2025-12-04T09:46:16.5878047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5878122Z outputs = self.mobilebert( 2025-12-04T09:46:16.5878447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5878550Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5878851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5878932Z layer_outputs = layer_module( 2025-12-04T09:46:16.5879261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5879353Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5879677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5879806Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5880125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.5880283Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5880675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5880786Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5880791Z 2025-12-04T09:46:16.5880910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5881147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5881219Z res = mod(**inputs) 2025-12-04T09:46:16.5881533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5881620Z outputs = self.mobilebert( 2025-12-04T09:46:16.5881929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5882008Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5882300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5882373Z layer_outputs = layer_module( 2025-12-04T09:46:16.5882665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5882763Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5883046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5883169Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5883453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5883545Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5883551Z 2025-12-04T09:46:16.5883654Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5883854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5883929Z res = mod(**inputs) 2025-12-04T09:46:16.5884209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5884284Z outputs = self.mobilebert( 2025-12-04T09:46:16.5884573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5884648Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5884939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5885012Z layer_outputs = layer_module( 2025-12-04T09:46:16.5885315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5885445Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5885730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5885865Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5886147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5886259Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5886263Z 2025-12-04T09:46:16.5886374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5886571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5886636Z res = mod(**inputs) 2025-12-04T09:46:16.5886948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5887020Z outputs = self.mobilebert( 2025-12-04T09:46:16.5887309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5887387Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5887670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5887751Z layer_outputs = layer_module( 2025-12-04T09:46:16.5888032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5888134Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5888415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5888547Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5888834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5888921Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5888925Z 2025-12-04T09:46:16.5889037Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5889236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5889301Z res = mod(**inputs) 2025-12-04T09:46:16.5889589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5889659Z outputs = self.mobilebert( 2025-12-04T09:46:16.5889941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5890024Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5890306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5890386Z layer_outputs = layer_module( 2025-12-04T09:46:16.5890668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5890763Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5891053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5891179Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5891469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5891610Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5891913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5892013Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5892016Z 2025-12-04T09:46:16.5892145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5892345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5892419Z res = mod(**inputs) 2025-12-04T09:46:16.5892704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5892782Z outputs = self.mobilebert( 2025-12-04T09:46:16.5893064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5893159Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5893458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5893528Z layer_outputs = layer_module( 2025-12-04T09:46:16.5893825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5893921Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5894220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5894345Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5894661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5894757Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5894769Z 2025-12-04T09:46:16.5894873Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5895073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5895145Z res = mod(**inputs) 2025-12-04T09:46:16.5895438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5895510Z outputs = self.mobilebert( 2025-12-04T09:46:16.5895806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5895878Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5896177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5896249Z layer_outputs = layer_module( 2025-12-04T09:46:16.5896536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5896641Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5896932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5897050Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5897340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5897453Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5897456Z 2025-12-04T09:46:16.5897570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5897771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5897836Z res = mod(**inputs) 2025-12-04T09:46:16.5898166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5898257Z outputs = self.mobilebert( 2025-12-04T09:46:16.5898561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5898639Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5898928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5899007Z layer_outputs = layer_module( 2025-12-04T09:46:16.5899284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5899385Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5899663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5899807Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5900085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5900168Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5900171Z 2025-12-04T09:46:16.5900270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5900469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5900532Z res = mod(**inputs) 2025-12-04T09:46:16.5900809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5900881Z outputs = self.mobilebert( 2025-12-04T09:46:16.5901166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5901251Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5901527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5901604Z layer_outputs = layer_module( 2025-12-04T09:46:16.5901885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5901978Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5902265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5902390Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5902681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5902811Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5903081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5903177Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5903182Z 2025-12-04T09:46:16.5903282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5903473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5903546Z res = mod(**inputs) 2025-12-04T09:46:16.5903815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5903891Z outputs = self.mobilebert( 2025-12-04T09:46:16.5904180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5904271Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5904552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5904622Z layer_outputs = layer_module( 2025-12-04T09:46:16.5904928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5905023Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5905307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5905425Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5905705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5905809Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5905821Z 2025-12-04T09:46:16.5905934Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5906134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5906206Z res = mod(**inputs) 2025-12-04T09:46:16.5906492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5906566Z outputs = self.mobilebert( 2025-12-04T09:46:16.5906858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5906932Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5907222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5907296Z layer_outputs = layer_module( 2025-12-04T09:46:16.5907581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5907683Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5907968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5908079Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5908372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5908484Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5908487Z 2025-12-04T09:46:16.5908598Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5908797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5908866Z res = mod(**inputs) 2025-12-04T09:46:16.5909161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5909236Z outputs = self.mobilebert( 2025-12-04T09:46:16.5909530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5909607Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5909899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5909985Z layer_outputs = layer_module( 2025-12-04T09:46:16.5910292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5910393Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5910729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5910883Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5911188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5911296Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5911300Z 2025-12-04T09:46:16.5911411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5911630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5911701Z res = mod(**inputs) 2025-12-04T09:46:16.5912017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5912092Z outputs = self.mobilebert( 2025-12-04T09:46:16.5912403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5912512Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5912817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5912893Z layer_outputs = layer_module( 2025-12-04T09:46:16.5913217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5913316Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5913634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5913769Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5914088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5914230Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5914538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5914644Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5914647Z 2025-12-04T09:46:16.5914758Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5914970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5915048Z res = mod(**inputs) 2025-12-04T09:46:16.5915363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5915447Z outputs = self.mobilebert( 2025-12-04T09:46:16.5915755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5915837Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5916152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5916229Z layer_outputs = layer_module( 2025-12-04T09:46:16.5916546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5916686Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5916994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5917090Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5917094Z 2025-12-04T09:46:16.5917204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5917435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5917534Z res = mod(**inputs) 2025-12-04T09:46:16.5917840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5917926Z outputs = self.mobilebert( 2025-12-04T09:46:16.5918264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5918347Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5918667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5918745Z layer_outputs = layer_module( 2025-12-04T09:46:16.5919060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.5919229Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.5919535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5919664Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5919668Z 2025-12-04T09:46:16.5919782Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5919996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5920073Z res = mod(**inputs) 2025-12-04T09:46:16.5920451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5920548Z outputs = self.mobilebert( 2025-12-04T09:46:16.5920869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5920956Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5921274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5921353Z layer_outputs = layer_module( 2025-12-04T09:46:16.5921662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5921845Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5922151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.5922262Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.5922267Z 2025-12-04T09:46:16.5922378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5922595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5922676Z res = mod(**inputs) 2025-12-04T09:46:16.5922982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5923068Z outputs = self.mobilebert( 2025-12-04T09:46:16.5923379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5923460Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5923777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5923855Z layer_outputs = layer_module( 2025-12-04T09:46:16.5924174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5924367Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5924676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.5924840Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.5925163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5925267Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5925277Z 2025-12-04T09:46:16.5925390Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5925607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5925686Z res = mod(**inputs) 2025-12-04T09:46:16.5926001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5926100Z outputs = self.mobilebert( 2025-12-04T09:46:16.5926414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5926497Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5926809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5926887Z layer_outputs = layer_module( 2025-12-04T09:46:16.5927194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5927372Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5927674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5927818Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5928126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.5928220Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5928224Z 2025-12-04T09:46:16.5928342Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5928558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5928629Z res = mod(**inputs) 2025-12-04T09:46:16.5928941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5929018Z outputs = self.mobilebert( 2025-12-04T09:46:16.5929332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5929412Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5929721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5929809Z layer_outputs = layer_module( 2025-12-04T09:46:16.5930117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.5930304Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.5930600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.5930731Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.5931036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.5931166Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5931513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5931641Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5931645Z 2025-12-04T09:46:16.5931756Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5931996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5932071Z res = mod(**inputs) 2025-12-04T09:46:16.5932387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5932473Z outputs = self.mobilebert( 2025-12-04T09:46:16.5932786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5932874Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5933202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5933282Z layer_outputs = layer_module( 2025-12-04T09:46:16.5933597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5933773Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5934105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5934220Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5934517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5934619Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5934623Z 2025-12-04T09:46:16.5934736Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5934954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5935033Z res = mod(**inputs) 2025-12-04T09:46:16.5935337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5935424Z outputs = self.mobilebert( 2025-12-04T09:46:16.5935742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5935821Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5936134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5936211Z layer_outputs = layer_module( 2025-12-04T09:46:16.5936537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5936636Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5936940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5937028Z self_outputs = self.self( 2025-12-04T09:46:16.5937344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.5937424Z self.value(value_tensor) 2025-12-04T09:46:16.5937437Z 2025-12-04T09:46:16.5937547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5937763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5937840Z res = mod(**inputs) 2025-12-04T09:46:16.5938171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5938265Z outputs = self.mobilebert( 2025-12-04T09:46:16.5938589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5938671Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5939010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5939090Z layer_outputs = layer_module( 2025-12-04T09:46:16.5939397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5939577Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5939896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.5940038Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.5940360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.5940450Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.5940455Z 2025-12-04T09:46:16.5940575Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5940793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5940866Z res = mod(**inputs) 2025-12-04T09:46:16.5941196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5941274Z outputs = self.mobilebert( 2025-12-04T09:46:16.5941605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5941689Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5942007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5942094Z layer_outputs = layer_module( 2025-12-04T09:46:16.5942413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.5942588Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.5942917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.5943037Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.5943357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.5943456Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.5943771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5943879Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5943882Z 2025-12-04T09:46:16.5943994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5944221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5944292Z res = mod(**inputs) 2025-12-04T09:46:16.5944603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5944689Z outputs = self.mobilebert( 2025-12-04T09:46:16.5945003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5945090Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5946528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5946624Z layer_outputs = layer_module( 2025-12-04T09:46:16.5946991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5947299Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5947620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5947709Z self_outputs = self.self( 2025-12-04T09:46:16.5948014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.5948103Z self.query(query_tensor) 2025-12-04T09:46:16.5948107Z 2025-12-04T09:46:16.5948222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5948486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5948569Z res = mod(**inputs) 2025-12-04T09:46:16.5948875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5948962Z outputs = self.mobilebert( 2025-12-04T09:46:16.5949273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5949356Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5949674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5949751Z layer_outputs = layer_module( 2025-12-04T09:46:16.5950061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5950167Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5950475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.5950564Z self_outputs = self.self( 2025-12-04T09:46:16.5950873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.5950949Z self.key(key_tensor) 2025-12-04T09:46:16.5950953Z 2025-12-04T09:46:16.5951057Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5951147Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.5951270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5951494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5951563Z res = mod(**inputs) 2025-12-04T09:46:16.5951871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5951950Z outputs = self.mobilebert( 2025-12-04T09:46:16.5952250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5952339Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5952638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5952724Z layer_outputs = layer_module( 2025-12-04T09:46:16.5953034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5953126Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5953469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5953610Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5953968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.5954062Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5954066Z 2025-12-04T09:46:16.5954213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5954441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5954513Z res = mod(**inputs) 2025-12-04T09:46:16.5954834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5954917Z outputs = self.mobilebert( 2025-12-04T09:46:16.5955226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5955333Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5955636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5955713Z layer_outputs = layer_module( 2025-12-04T09:46:16.5956037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.5956131Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.5956450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.5956594Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.5956912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.5957062Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5957375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5957478Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5957490Z 2025-12-04T09:46:16.5957605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5957823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5957903Z res = mod(**inputs) 2025-12-04T09:46:16.5958221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5958299Z outputs = self.mobilebert( 2025-12-04T09:46:16.5958618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5958701Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5959019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5959097Z layer_outputs = layer_module( 2025-12-04T09:46:16.5959417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5959531Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5959850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5959978Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5960304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.5960456Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.5960466Z 2025-12-04T09:46:16.5960612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5960848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5960920Z res = mod(**inputs) 2025-12-04T09:46:16.5961249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5961329Z outputs = self.mobilebert( 2025-12-04T09:46:16.5961648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5961727Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5962038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5962124Z layer_outputs = layer_module( 2025-12-04T09:46:16.5962450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5962578Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5962886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.5963009Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.5963316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.5963436Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.5963440Z 2025-12-04T09:46:16.5992188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5992614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5992704Z res = mod(**inputs) 2025-12-04T09:46:16.5993110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5993213Z outputs = self.mobilebert( 2025-12-04T09:46:16.5993538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5993642Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5993959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5994045Z layer_outputs = layer_module( 2025-12-04T09:46:16.5994369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5994478Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5994789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5994941Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5995243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.5995350Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.5995359Z 2025-12-04T09:46:16.5995484Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5995724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5995800Z res = mod(**inputs) 2025-12-04T09:46:16.5996115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5996211Z outputs = self.mobilebert( 2025-12-04T09:46:16.5996664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.5996789Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.5997108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.5997189Z layer_outputs = layer_module( 2025-12-04T09:46:16.5997547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.5997659Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.5997968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.5998123Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.5998433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.5998617Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.5998927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.5999032Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.5999037Z 2025-12-04T09:46:16.5999167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.5999393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.5999475Z res = mod(**inputs) 2025-12-04T09:46:16.5999785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.5999869Z outputs = self.mobilebert( 2025-12-04T09:46:16.6000182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6000281Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6000684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6000772Z layer_outputs = layer_module( 2025-12-04T09:46:16.6001093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6001199Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6001516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6001647Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6001957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6002063Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6002069Z 2025-12-04T09:46:16.6002191Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6002423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6002496Z res = mod(**inputs) 2025-12-04T09:46:16.6002807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6002896Z outputs = self.mobilebert( 2025-12-04T09:46:16.6003208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6003289Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6003607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6003688Z layer_outputs = layer_module( 2025-12-04T09:46:16.6004030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6004155Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6004464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6004615Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6004924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6005057Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6005061Z 2025-12-04T09:46:16.6005177Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6005397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6005474Z res = mod(**inputs) 2025-12-04T09:46:16.6005803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6005885Z outputs = self.mobilebert( 2025-12-04T09:46:16.6006196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6006276Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6006586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6006662Z layer_outputs = layer_module( 2025-12-04T09:46:16.6006965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6007074Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6007383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6007528Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6007835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6007927Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6007931Z 2025-12-04T09:46:16.6008049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6008266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6008343Z res = mod(**inputs) 2025-12-04T09:46:16.6008645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6008723Z outputs = self.mobilebert( 2025-12-04T09:46:16.6009036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6009119Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6009427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6009511Z layer_outputs = layer_module( 2025-12-04T09:46:16.6009822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6009930Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6010236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6010374Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6010721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6010877Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6011194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6011307Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6011328Z 2025-12-04T09:46:16.6011440Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6011661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6011729Z res = mod(**inputs) 2025-12-04T09:46:16.6012028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6012110Z outputs = self.mobilebert( 2025-12-04T09:46:16.6012407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6012513Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6012819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6012892Z layer_outputs = layer_module( 2025-12-04T09:46:16.6013204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6013301Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6013608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6013729Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6014027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6014126Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6014132Z 2025-12-04T09:46:16.6014239Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6014461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6014533Z res = mod(**inputs) 2025-12-04T09:46:16.6014854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6014941Z outputs = self.mobilebert( 2025-12-04T09:46:16.6015262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6015341Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6015670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6015746Z layer_outputs = layer_module( 2025-12-04T09:46:16.6016067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6016169Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6016483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6016612Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6016926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6017056Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6017059Z 2025-12-04T09:46:16.6017171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6017387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6017486Z res = mod(**inputs) 2025-12-04T09:46:16.6017816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6017896Z outputs = self.mobilebert( 2025-12-04T09:46:16.6018238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6018319Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6018641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6018719Z layer_outputs = layer_module( 2025-12-04T09:46:16.6019037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6019149Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6019467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6019645Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6019956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6020051Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6020055Z 2025-12-04T09:46:16.6020174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6020390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6020460Z res = mod(**inputs) 2025-12-04T09:46:16.6020771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6020848Z outputs = self.mobilebert( 2025-12-04T09:46:16.6021162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6021245Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6021551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6021638Z layer_outputs = layer_module( 2025-12-04T09:46:16.6021945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6022053Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6022370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6022509Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6022831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6022964Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6023272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6023371Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6023374Z 2025-12-04T09:46:16.6023483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6023701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6023769Z res = mod(**inputs) 2025-12-04T09:46:16.6024072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6024158Z outputs = self.mobilebert( 2025-12-04T09:46:16.6024487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6024596Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6024909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6024986Z layer_outputs = layer_module( 2025-12-04T09:46:16.6025327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6025464Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6025780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6025872Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6025876Z 2025-12-04T09:46:16.6025986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6026215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6026317Z res = mod(**inputs) 2025-12-04T09:46:16.6026632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6026716Z outputs = self.mobilebert( 2025-12-04T09:46:16.6027033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6027120Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6027437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6027514Z layer_outputs = layer_module( 2025-12-04T09:46:16.6027841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6027978Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6028304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6028431Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6028435Z 2025-12-04T09:46:16.6028551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6028778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6028849Z res = mod(**inputs) 2025-12-04T09:46:16.6029168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6029254Z outputs = self.mobilebert( 2025-12-04T09:46:16.6029566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6029655Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6029970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6030049Z layer_outputs = layer_module( 2025-12-04T09:46:16.6030373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6030555Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6030881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6030985Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6030989Z 2025-12-04T09:46:16.6031102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6031332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6031421Z res = mod(**inputs) 2025-12-04T09:46:16.6031745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6031830Z outputs = self.mobilebert( 2025-12-04T09:46:16.6032152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6032241Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6032547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6032623Z layer_outputs = layer_module( 2025-12-04T09:46:16.6032937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6033115Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6033448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6033587Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6033898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6034005Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6034009Z 2025-12-04T09:46:16.6034118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6034334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6034403Z res = mod(**inputs) 2025-12-04T09:46:16.6034707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6034791Z outputs = self.mobilebert( 2025-12-04T09:46:16.6035099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6035180Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6035498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6035575Z layer_outputs = layer_module( 2025-12-04T09:46:16.6035889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6036063Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6036382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6036526Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6036836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6036936Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6036940Z 2025-12-04T09:46:16.6037053Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6037272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6037350Z res = mod(**inputs) 2025-12-04T09:46:16.6037655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6037732Z outputs = self.mobilebert( 2025-12-04T09:46:16.6038047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6038124Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6038461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6038561Z layer_outputs = layer_module( 2025-12-04T09:46:16.6038874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6039073Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6039379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6039520Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6039828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6039960Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6040293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6040488Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6040495Z 2025-12-04T09:46:16.6040621Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6040844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6040914Z res = mod(**inputs) 2025-12-04T09:46:16.6041229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6041309Z outputs = self.mobilebert( 2025-12-04T09:46:16.6041617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6041708Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6042027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6042117Z layer_outputs = layer_module( 2025-12-04T09:46:16.6042426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6042608Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6042938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6043062Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6043389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6043482Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6043487Z 2025-12-04T09:46:16.6043606Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6043835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6043903Z res = mod(**inputs) 2025-12-04T09:46:16.6044230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6044306Z outputs = self.mobilebert( 2025-12-04T09:46:16.6044623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6044705Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6045020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6045094Z layer_outputs = layer_module( 2025-12-04T09:46:16.6045428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6045546Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6045856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6045935Z self_outputs = self.self( 2025-12-04T09:46:16.6046264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6046354Z self.value(value_tensor) 2025-12-04T09:46:16.6046358Z 2025-12-04T09:46:16.6046473Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6046693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6046763Z res = mod(**inputs) 2025-12-04T09:46:16.6047325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6047488Z outputs = self.mobilebert( 2025-12-04T09:46:16.6047800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6047884Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6048204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6048282Z layer_outputs = layer_module( 2025-12-04T09:46:16.6048597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6048780Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6049094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6049232Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6049544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6049646Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6049651Z 2025-12-04T09:46:16.6049766Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6049987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6050067Z res = mod(**inputs) 2025-12-04T09:46:16.6050377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6050456Z outputs = self.mobilebert( 2025-12-04T09:46:16.6050776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6050859Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6051183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6051261Z layer_outputs = layer_module( 2025-12-04T09:46:16.6051574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6051762Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6052070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6052199Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6052509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6052640Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6052956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6053083Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6053087Z 2025-12-04T09:46:16.6053205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6053448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6053522Z res = mod(**inputs) 2025-12-04T09:46:16.6053833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6053909Z outputs = self.mobilebert( 2025-12-04T09:46:16.6054217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6054305Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6054630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6054715Z layer_outputs = layer_module( 2025-12-04T09:46:16.6055021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6055119Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6055432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6055510Z self_outputs = self.self( 2025-12-04T09:46:16.6055815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6055904Z self.query(query_tensor) 2025-12-04T09:46:16.6055907Z 2025-12-04T09:46:16.6056020Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6056245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6056315Z res = mod(**inputs) 2025-12-04T09:46:16.6056612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6056693Z outputs = self.mobilebert( 2025-12-04T09:46:16.6056994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6057075Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6057391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6057467Z layer_outputs = layer_module( 2025-12-04T09:46:16.6057778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6057873Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6058175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6058260Z self_outputs = self.self( 2025-12-04T09:46:16.6058577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6058658Z self.key(key_tensor) 2025-12-04T09:46:16.6058662Z 2025-12-04T09:46:16.6058752Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6058840Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6058961Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6059176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6059254Z res = mod(**inputs) 2025-12-04T09:46:16.6059577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6059677Z outputs = self.mobilebert( 2025-12-04T09:46:16.6059994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6060129Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6060449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6060532Z layer_outputs = layer_module( 2025-12-04T09:46:16.6060846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6060948Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6061253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6061414Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6061727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6061820Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6061824Z 2025-12-04T09:46:16.6061945Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6062165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6062235Z res = mod(**inputs) 2025-12-04T09:46:16.6062546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6062623Z outputs = self.mobilebert( 2025-12-04T09:46:16.6062936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6063026Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6063332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6063416Z layer_outputs = layer_module( 2025-12-04T09:46:16.6063722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6063817Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6064131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6064267Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6064583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6064725Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6065036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6065148Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6065152Z 2025-12-04T09:46:16.6065266Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6065493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6065564Z res = mod(**inputs) 2025-12-04T09:46:16.6065871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6065958Z outputs = self.mobilebert( 2025-12-04T09:46:16.6066273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6066370Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6066698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6066798Z layer_outputs = layer_module( 2025-12-04T09:46:16.6067142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6067250Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6067570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6067704Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6068022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6068122Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6068144Z 2025-12-04T09:46:16.6068261Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6068481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6068558Z res = mod(**inputs) 2025-12-04T09:46:16.6068877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6068955Z outputs = self.mobilebert( 2025-12-04T09:46:16.6069287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6069366Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6069693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6069768Z layer_outputs = layer_module( 2025-12-04T09:46:16.6070091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6070204Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6070527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6070659Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6070976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6071101Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6071104Z 2025-12-04T09:46:16.6071227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6071446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6071516Z res = mod(**inputs) 2025-12-04T09:46:16.6071843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6071923Z outputs = self.mobilebert( 2025-12-04T09:46:16.6072244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6072324Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6072645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6072735Z layer_outputs = layer_module( 2025-12-04T09:46:16.6073049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6073157Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6073494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6073668Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6074001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6074094Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6074116Z 2025-12-04T09:46:16.6074235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6074450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6074520Z res = mod(**inputs) 2025-12-04T09:46:16.6074837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6074913Z outputs = self.mobilebert( 2025-12-04T09:46:16.6075230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6075338Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6075657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6075738Z layer_outputs = layer_module( 2025-12-04T09:46:16.6076104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6076209Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6076520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6076657Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6076977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6077115Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6077424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6077532Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6077536Z 2025-12-04T09:46:16.6077648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6077869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6077938Z res = mod(**inputs) 2025-12-04T09:46:16.6078241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6078324Z outputs = self.mobilebert( 2025-12-04T09:46:16.6078628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6078709Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6079022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6079099Z layer_outputs = layer_module( 2025-12-04T09:46:16.6079410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6079511Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6079814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6079945Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6080253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6080428Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6080462Z 2025-12-04T09:46:16.6080601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6080818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6080899Z res = mod(**inputs) 2025-12-04T09:46:16.6081223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6081304Z outputs = self.mobilebert( 2025-12-04T09:46:16.6081622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6081702Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6082015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6082093Z layer_outputs = layer_module( 2025-12-04T09:46:16.6082427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6082542Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6082853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6082983Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6083296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6083420Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6083424Z 2025-12-04T09:46:16.6083542Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6083761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6083831Z res = mod(**inputs) 2025-12-04T09:46:16.6084153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6084232Z outputs = self.mobilebert( 2025-12-04T09:46:16.6084550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6084629Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6084962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6085046Z layer_outputs = layer_module( 2025-12-04T09:46:16.6085355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6085466Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6085802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6085941Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6086260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6086356Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6086360Z 2025-12-04T09:46:16.6086479Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6086697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6086767Z res = mod(**inputs) 2025-12-04T09:46:16.6087084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6087159Z outputs = self.mobilebert( 2025-12-04T09:46:16.6087497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6087602Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6087908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6087992Z layer_outputs = layer_module( 2025-12-04T09:46:16.6088314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6088419Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6088732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6088866Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6089179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6089333Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6089645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6089753Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6089759Z 2025-12-04T09:46:16.6089870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6090084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6090163Z res = mod(**inputs) 2025-12-04T09:46:16.6090468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6090553Z outputs = self.mobilebert( 2025-12-04T09:46:16.6090860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6090943Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6091254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6091331Z layer_outputs = layer_module( 2025-12-04T09:46:16.6091641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6091742Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6092047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6092177Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6092481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6092576Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6092588Z 2025-12-04T09:46:16.6092698Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6092912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6092987Z res = mod(**inputs) 2025-12-04T09:46:16.6093292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6093368Z outputs = self.mobilebert( 2025-12-04T09:46:16.6093680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6093756Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6094070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6094146Z layer_outputs = layer_module( 2025-12-04T09:46:16.6094472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6094620Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6094952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6095076Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6095386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6095509Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6095513Z 2025-12-04T09:46:16.6095629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6095843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6095933Z res = mod(**inputs) 2025-12-04T09:46:16.6096247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6096324Z outputs = self.mobilebert( 2025-12-04T09:46:16.6096644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6096722Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6097030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6097116Z layer_outputs = layer_module( 2025-12-04T09:46:16.6097424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6097538Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6097867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6098005Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6098322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6098417Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6098420Z 2025-12-04T09:46:16.6098531Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6098753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6098823Z res = mod(**inputs) 2025-12-04T09:46:16.6099137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6099214Z outputs = self.mobilebert( 2025-12-04T09:46:16.6099545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6099635Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6099945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6100030Z layer_outputs = layer_module( 2025-12-04T09:46:16.6100338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6100438Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6100773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6100908Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6101277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6101445Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6101756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6101881Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6101886Z 2025-12-04T09:46:16.6102001Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6102216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6102293Z res = mod(**inputs) 2025-12-04T09:46:16.6102598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6102683Z outputs = self.mobilebert( 2025-12-04T09:46:16.6102991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6103094Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6103415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6103492Z layer_outputs = layer_module( 2025-12-04T09:46:16.6103800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6103945Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6104279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6104378Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6104382Z 2025-12-04T09:46:16.6104493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6104713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6104795Z res = mod(**inputs) 2025-12-04T09:46:16.6105104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6105186Z outputs = self.mobilebert( 2025-12-04T09:46:16.6105499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6105576Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6105897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6105973Z layer_outputs = layer_module( 2025-12-04T09:46:16.6106293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6106429Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6106746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6106877Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6106881Z 2025-12-04T09:46:16.6106992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6107212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6107290Z res = mod(**inputs) 2025-12-04T09:46:16.6107606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6107690Z outputs = self.mobilebert( 2025-12-04T09:46:16.6108027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6108130Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6108466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6108544Z layer_outputs = layer_module( 2025-12-04T09:46:16.6108910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6109090Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6109424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6109538Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6109541Z 2025-12-04T09:46:16.6109653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6109871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6109971Z res = mod(**inputs) 2025-12-04T09:46:16.6110305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6110390Z outputs = self.mobilebert( 2025-12-04T09:46:16.6110726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6110806Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6111144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6111220Z layer_outputs = layer_module( 2025-12-04T09:46:16.6111539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6111715Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6112050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6112192Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6112533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6112640Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6112644Z 2025-12-04T09:46:16.6112758Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6112973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6113047Z res = mod(**inputs) 2025-12-04T09:46:16.6113361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6113438Z outputs = self.mobilebert( 2025-12-04T09:46:16.6113770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6113861Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6114184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6114259Z layer_outputs = layer_module( 2025-12-04T09:46:16.6114600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6114778Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6115099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6115240Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6115590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6115702Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6115706Z 2025-12-04T09:46:16.6115825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6116057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6116129Z res = mod(**inputs) 2025-12-04T09:46:16.6116457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6116535Z outputs = self.mobilebert( 2025-12-04T09:46:16.6116863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6116941Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6117264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6117373Z layer_outputs = layer_module( 2025-12-04T09:46:16.6117697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6117878Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6118194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6118331Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6118658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6118792Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6119120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6119224Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6119228Z 2025-12-04T09:46:16.6119344Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6119577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6119648Z res = mod(**inputs) 2025-12-04T09:46:16.6119970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6120058Z outputs = self.mobilebert( 2025-12-04T09:46:16.6120457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6120556Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6120878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6120961Z layer_outputs = layer_module( 2025-12-04T09:46:16.6121277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6121459Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6121787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6121911Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6122230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6122334Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6122338Z 2025-12-04T09:46:16.6122475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6122725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6122796Z res = mod(**inputs) 2025-12-04T09:46:16.6123133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6123225Z outputs = self.mobilebert( 2025-12-04T09:46:16.6123546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6123627Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6123960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6124038Z layer_outputs = layer_module( 2025-12-04T09:46:16.6124361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6124488Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6124795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6124884Z self_outputs = self.self( 2025-12-04T09:46:16.6125200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6125281Z self.value(value_tensor) 2025-12-04T09:46:16.6125290Z 2025-12-04T09:46:16.6125401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6125614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6125694Z res = mod(**inputs) 2025-12-04T09:46:16.6126007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6126085Z outputs = self.mobilebert( 2025-12-04T09:46:16.6126400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6126478Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6126802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6126880Z layer_outputs = layer_module( 2025-12-04T09:46:16.6127194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6127379Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6127692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6127826Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6128131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6128221Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6128225Z 2025-12-04T09:46:16.6128347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6128561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6128631Z res = mod(**inputs) 2025-12-04T09:46:16.6128945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6129021Z outputs = self.mobilebert( 2025-12-04T09:46:16.6129334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6129434Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6129760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6129846Z layer_outputs = layer_module( 2025-12-04T09:46:16.6130175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6130362Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6130670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6130793Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6131110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6131203Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6131536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6131645Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6131649Z 2025-12-04T09:46:16.6131760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6131982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6132051Z res = mod(**inputs) 2025-12-04T09:46:16.6132353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6132438Z outputs = self.mobilebert( 2025-12-04T09:46:16.6132741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6132825Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6133137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6133213Z layer_outputs = layer_module( 2025-12-04T09:46:16.6133527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6133623Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6133929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6134014Z self_outputs = self.self( 2025-12-04T09:46:16.6134321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6134405Z self.query(query_tensor) 2025-12-04T09:46:16.6134409Z 2025-12-04T09:46:16.6134523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6134740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6134823Z res = mod(**inputs) 2025-12-04T09:46:16.6135127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6135212Z outputs = self.mobilebert( 2025-12-04T09:46:16.6135518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6135596Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6135909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6135986Z layer_outputs = layer_module( 2025-12-04T09:46:16.6136291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6136420Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6136746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6136831Z self_outputs = self.self( 2025-12-04T09:46:16.6137160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6137237Z self.key(key_tensor) 2025-12-04T09:46:16.6137241Z 2025-12-04T09:46:16.6137337Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6137426Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6137546Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6137777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6137856Z res = mod(**inputs) 2025-12-04T09:46:16.6138168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6138268Z outputs = self.mobilebert( 2025-12-04T09:46:16.6138602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6138680Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6139027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6139103Z layer_outputs = layer_module( 2025-12-04T09:46:16.6139421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6139522Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6139862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6140010Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6140334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6140426Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6140430Z 2025-12-04T09:46:16.6140550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6140771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6140842Z res = mod(**inputs) 2025-12-04T09:46:16.6141181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6141258Z outputs = self.mobilebert( 2025-12-04T09:46:16.6141590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6141672Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6141995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6142079Z layer_outputs = layer_module( 2025-12-04T09:46:16.6142409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6142509Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6142827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6142962Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6143297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6143455Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6143783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6143894Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6143898Z 2025-12-04T09:46:16.6144027Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6144252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6144323Z res = mod(**inputs) 2025-12-04T09:46:16.6144637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6144722Z outputs = self.mobilebert( 2025-12-04T09:46:16.6145037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6145124Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6145447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6145525Z layer_outputs = layer_module( 2025-12-04T09:46:16.6145839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6145943Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6146260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6146390Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6146706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6146805Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6146810Z 2025-12-04T09:46:16.6146923Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6147291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6147373Z res = mod(**inputs) 2025-12-04T09:46:16.6147698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6147787Z outputs = self.mobilebert( 2025-12-04T09:46:16.6148125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6148204Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6148521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6148598Z layer_outputs = layer_module( 2025-12-04T09:46:16.6148941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6149050Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6149358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6149492Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6149804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6149928Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6149941Z 2025-12-04T09:46:16.6150056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6150274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6150354Z res = mod(**inputs) 2025-12-04T09:46:16.6150714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6150829Z outputs = self.mobilebert( 2025-12-04T09:46:16.6151156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6151273Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6151592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6151671Z layer_outputs = layer_module( 2025-12-04T09:46:16.6151978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6152094Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6152405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6152575Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6152891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6152984Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6152991Z 2025-12-04T09:46:16.6153112Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6153329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6153400Z res = mod(**inputs) 2025-12-04T09:46:16.6153715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6153792Z outputs = self.mobilebert( 2025-12-04T09:46:16.6154106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6154186Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6154495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6154579Z layer_outputs = layer_module( 2025-12-04T09:46:16.6154887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6154987Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6155302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6155439Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6155752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6155889Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6156196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6156304Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6156308Z 2025-12-04T09:46:16.6156422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6156646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6156714Z res = mod(**inputs) 2025-12-04T09:46:16.6157018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6157102Z outputs = self.mobilebert( 2025-12-04T09:46:16.6157405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6157513Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6157835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6157912Z layer_outputs = layer_module( 2025-12-04T09:46:16.6158242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6158347Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6158655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6158785Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6159093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6159194Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6159218Z 2025-12-04T09:46:16.6159332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6159548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6159626Z res = mod(**inputs) 2025-12-04T09:46:16.6159928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6160012Z outputs = self.mobilebert( 2025-12-04T09:46:16.6160319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6160467Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6160784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6160861Z layer_outputs = layer_module( 2025-12-04T09:46:16.6161172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6161288Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6161592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6161724Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6162033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6162158Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6162163Z 2025-12-04T09:46:16.6162286Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6162502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6162581Z res = mod(**inputs) 2025-12-04T09:46:16.6162893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6162974Z outputs = self.mobilebert( 2025-12-04T09:46:16.6163289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6163371Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6163678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6163762Z layer_outputs = layer_module( 2025-12-04T09:46:16.6164067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6164177Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6164508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6164668Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6164983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6165092Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6165096Z 2025-12-04T09:46:16.6165219Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6165434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6165504Z res = mod(**inputs) 2025-12-04T09:46:16.6165818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6165896Z outputs = self.mobilebert( 2025-12-04T09:46:16.6166209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6166307Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6166616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6166700Z layer_outputs = layer_module( 2025-12-04T09:46:16.6167015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6167116Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6167434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6167569Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6167888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6168023Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6168336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6168445Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6168448Z 2025-12-04T09:46:16.6168563Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6168789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6168859Z res = mod(**inputs) 2025-12-04T09:46:16.6169166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6169253Z outputs = self.mobilebert( 2025-12-04T09:46:16.6169564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6169645Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6169964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6170040Z layer_outputs = layer_module( 2025-12-04T09:46:16.6170372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6170472Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6170782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6170912Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6171224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6171345Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6171401Z 2025-12-04T09:46:16.6171515Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6171730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6171808Z res = mod(**inputs) 2025-12-04T09:46:16.6172131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6172211Z outputs = self.mobilebert( 2025-12-04T09:46:16.6172534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6172612Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6172933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6173010Z layer_outputs = layer_module( 2025-12-04T09:46:16.6173337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6173449Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6173757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6173884Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6174188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6174311Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6174314Z 2025-12-04T09:46:16.6174434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6174645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6174725Z res = mod(**inputs) 2025-12-04T09:46:16.6175024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6175103Z outputs = self.mobilebert( 2025-12-04T09:46:16.6175416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6175495Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6175801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6175887Z layer_outputs = layer_module( 2025-12-04T09:46:16.6176193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6176301Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6176607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6176746Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6177059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6177153Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6177156Z 2025-12-04T09:46:16.6177273Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6177489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6177558Z res = mod(**inputs) 2025-12-04T09:46:16.6177870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6177946Z outputs = self.mobilebert( 2025-12-04T09:46:16.6178275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6178390Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6178706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6178812Z layer_outputs = layer_module( 2025-12-04T09:46:16.6179121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6179222Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6179537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6179672Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6179991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6180152Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6180460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6180570Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6180573Z 2025-12-04T09:46:16.6180685Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6180911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6180979Z res = mod(**inputs) 2025-12-04T09:46:16.6181287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6181369Z outputs = self.mobilebert( 2025-12-04T09:46:16.6181678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6181759Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6182074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6182151Z layer_outputs = layer_module( 2025-12-04T09:46:16.6182463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6182598Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6182905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6183005Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6183009Z 2025-12-04T09:46:16.6183121Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6183345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6183420Z res = mod(**inputs) 2025-12-04T09:46:16.6183726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6183810Z outputs = self.mobilebert( 2025-12-04T09:46:16.6184119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6184199Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6184515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6184592Z layer_outputs = layer_module( 2025-12-04T09:46:16.6184905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6185059Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6185390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6185523Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6185528Z 2025-12-04T09:46:16.6185658Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6185885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6185957Z res = mod(**inputs) 2025-12-04T09:46:16.6186265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6186353Z outputs = self.mobilebert( 2025-12-04T09:46:16.6186663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6186748Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6187092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6187167Z layer_outputs = layer_module( 2025-12-04T09:46:16.6187480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6187652Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6187957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6188069Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6188073Z 2025-12-04T09:46:16.6188183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6188407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6188482Z res = mod(**inputs) 2025-12-04T09:46:16.6188789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6188875Z outputs = self.mobilebert( 2025-12-04T09:46:16.6189183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6189261Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6189574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6189652Z layer_outputs = layer_module( 2025-12-04T09:46:16.6189963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6190135Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6190442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6190587Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6190908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6191017Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6191021Z 2025-12-04T09:46:16.6191132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6191347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6191424Z res = mod(**inputs) 2025-12-04T09:46:16.6191741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6191824Z outputs = self.mobilebert( 2025-12-04T09:46:16.6192158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6192600Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6192959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6193038Z layer_outputs = layer_module( 2025-12-04T09:46:16.6193362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6193543Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6193867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6194014Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6194343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6194440Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6194444Z 2025-12-04T09:46:16.6194563Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6194781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6194860Z res = mod(**inputs) 2025-12-04T09:46:16.6195180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6195257Z outputs = self.mobilebert( 2025-12-04T09:46:16.6195586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6195665Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6195988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6196079Z layer_outputs = layer_module( 2025-12-04T09:46:16.6196410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6196591Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6196921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6197056Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6197383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6197516Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6197833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6197935Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6197939Z 2025-12-04T09:46:16.6198050Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6198279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6198349Z res = mod(**inputs) 2025-12-04T09:46:16.6198662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6198739Z outputs = self.mobilebert( 2025-12-04T09:46:16.6199043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6199130Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6199459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6199555Z layer_outputs = layer_module( 2025-12-04T09:46:16.6199870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6200068Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6200480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6200612Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6200921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6201023Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6201028Z 2025-12-04T09:46:16.6201146Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6201402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6201473Z res = mod(**inputs) 2025-12-04T09:46:16.6201776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6201864Z outputs = self.mobilebert( 2025-12-04T09:46:16.6202171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6202256Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6202564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6202640Z layer_outputs = layer_module( 2025-12-04T09:46:16.6202954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6203053Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6203360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6203449Z self_outputs = self.self( 2025-12-04T09:46:16.6203756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6203846Z self.value(value_tensor) 2025-12-04T09:46:16.6203850Z 2025-12-04T09:46:16.6203962Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6204178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6204256Z res = mod(**inputs) 2025-12-04T09:46:16.6204562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6204641Z outputs = self.mobilebert( 2025-12-04T09:46:16.6204953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6205031Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6205347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6205424Z layer_outputs = layer_module( 2025-12-04T09:46:16.6205731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6205912Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6206222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6206386Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6206720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6206811Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6206816Z 2025-12-04T09:46:16.6206956Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6207176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6207256Z res = mod(**inputs) 2025-12-04T09:46:16.6207568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6207645Z outputs = self.mobilebert( 2025-12-04T09:46:16.6207965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6208046Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6208373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6208459Z layer_outputs = layer_module( 2025-12-04T09:46:16.6208778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6208957Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6209259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6209382Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6209703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6209800Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6210119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6210224Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6210228Z 2025-12-04T09:46:16.6210339Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6210565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6210638Z res = mod(**inputs) 2025-12-04T09:46:16.6210973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6211048Z outputs = self.mobilebert( 2025-12-04T09:46:16.6211345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6211429Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6211731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6211806Z layer_outputs = layer_module( 2025-12-04T09:46:16.6212113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6212206Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6212513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6212591Z self_outputs = self.self( 2025-12-04T09:46:16.6212898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6212983Z self.query(query_tensor) 2025-12-04T09:46:16.6212986Z 2025-12-04T09:46:16.6213099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6213345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6213433Z res = mod(**inputs) 2025-12-04T09:46:16.6213740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6213844Z outputs = self.mobilebert( 2025-12-04T09:46:16.6214153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6214231Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6214548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6214626Z layer_outputs = layer_module( 2025-12-04T09:46:16.6214939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6215054Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6215365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6215451Z self_outputs = self.self( 2025-12-04T09:46:16.6215780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6215851Z self.key(key_tensor) 2025-12-04T09:46:16.6215862Z 2025-12-04T09:46:16.6215950Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6216033Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6216150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6216362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6216430Z res = mod(**inputs) 2025-12-04T09:46:16.6216734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6216811Z outputs = self.mobilebert( 2025-12-04T09:46:16.6217124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6217203Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6217513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6217598Z layer_outputs = layer_module( 2025-12-04T09:46:16.6217905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6217997Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6218309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6218448Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6218765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6218857Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6218861Z 2025-12-04T09:46:16.6218974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6219202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6219272Z res = mod(**inputs) 2025-12-04T09:46:16.6219587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6219663Z outputs = self.mobilebert( 2025-12-04T09:46:16.6219969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6220079Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6220406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6220483Z layer_outputs = layer_module( 2025-12-04T09:46:16.6220816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6220910Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6221224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6221362Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6221673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6221825Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6222154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6222261Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6222264Z 2025-12-04T09:46:16.6222375Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6222590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6222669Z res = mod(**inputs) 2025-12-04T09:46:16.6222973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6223050Z outputs = self.mobilebert( 2025-12-04T09:46:16.6223361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6223441Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6223760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6223840Z layer_outputs = layer_module( 2025-12-04T09:46:16.6224145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6224269Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6224566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6224693Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6224992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6225084Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6225088Z 2025-12-04T09:46:16.6225210Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6225428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6225507Z res = mod(**inputs) 2025-12-04T09:46:16.6225813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6225891Z outputs = self.mobilebert( 2025-12-04T09:46:16.6226202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6226280Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6226587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6226674Z layer_outputs = layer_module( 2025-12-04T09:46:16.6227007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6227141Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6227449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6227590Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6227905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6228029Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6228033Z 2025-12-04T09:46:16.6228152Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6228369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6228438Z res = mod(**inputs) 2025-12-04T09:46:16.6228752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6228863Z outputs = self.mobilebert( 2025-12-04T09:46:16.6229180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6229266Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6229585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6229670Z layer_outputs = layer_module( 2025-12-04T09:46:16.6230007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6230111Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6230434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6230574Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6230898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6230988Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6230992Z 2025-12-04T09:46:16.6231106Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6231333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6231402Z res = mod(**inputs) 2025-12-04T09:46:16.6231737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6231821Z outputs = self.mobilebert( 2025-12-04T09:46:16.6232163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6232252Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6232590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6232667Z layer_outputs = layer_module( 2025-12-04T09:46:16.6233007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6233110Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6233437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6233572Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6233897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6234062Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6234403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6234509Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6234513Z 2025-12-04T09:46:16.6234642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6234861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6234938Z res = mod(**inputs) 2025-12-04T09:46:16.6235252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6235328Z outputs = self.mobilebert( 2025-12-04T09:46:16.6235672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6235774Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6236104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6236184Z layer_outputs = layer_module( 2025-12-04T09:46:16.6236505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6236619Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6236944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6237075Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6237396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6237492Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6237499Z 2025-12-04T09:46:16.6237623Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6237847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6237920Z res = mod(**inputs) 2025-12-04T09:46:16.6238248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6238330Z outputs = self.mobilebert( 2025-12-04T09:46:16.6238659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6238740Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6239063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6239151Z layer_outputs = layer_module( 2025-12-04T09:46:16.6239477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6239592Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6239932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6240060Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6240468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6240602Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6240607Z 2025-12-04T09:46:16.6240729Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6240948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6241020Z res = mod(**inputs) 2025-12-04T09:46:16.6241381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6241482Z outputs = self.mobilebert( 2025-12-04T09:46:16.6241815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6241924Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6242257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6242343Z layer_outputs = layer_module( 2025-12-04T09:46:16.6242673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6242778Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6243112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6243277Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6243589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6243683Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6243687Z 2025-12-04T09:46:16.6243798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6244023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6244094Z res = mod(**inputs) 2025-12-04T09:46:16.6244424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6244507Z outputs = self.mobilebert( 2025-12-04T09:46:16.6244839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6244926Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6245229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6245304Z layer_outputs = layer_module( 2025-12-04T09:46:16.6245636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6245738Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6246065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6246198Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6246496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6246640Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6246945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6247178Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6247194Z 2025-12-04T09:46:16.6247315Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6247529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6247608Z res = mod(**inputs) 2025-12-04T09:46:16.6247917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6247993Z outputs = self.mobilebert( 2025-12-04T09:46:16.6248353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6248438Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6248779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6248855Z layer_outputs = layer_module( 2025-12-04T09:46:16.6249195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6249307Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6249613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6249735Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6250058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6250152Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6250180Z 2025-12-04T09:46:16.6250302Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6250517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6250587Z res = mod(**inputs) 2025-12-04T09:46:16.6250905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6250991Z outputs = self.mobilebert( 2025-12-04T09:46:16.6251310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6251387Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6251692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6251775Z layer_outputs = layer_module( 2025-12-04T09:46:16.6252073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6252175Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6252492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6252613Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6252927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6253049Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6253052Z 2025-12-04T09:46:16.6253164Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6253387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6253459Z res = mod(**inputs) 2025-12-04T09:46:16.6253774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6253852Z outputs = self.mobilebert( 2025-12-04T09:46:16.6254167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6254251Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6254549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6254630Z layer_outputs = layer_module( 2025-12-04T09:46:16.6254932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6255034Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6255369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6255526Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6255835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6255965Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6255969Z 2025-12-04T09:46:16.6256080Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6256300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6256370Z res = mod(**inputs) 2025-12-04T09:46:16.6256673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6256756Z outputs = self.mobilebert( 2025-12-04T09:46:16.6257062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6257169Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6257474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6257553Z layer_outputs = layer_module( 2025-12-04T09:46:16.6257868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6257971Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6258279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6258422Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6258729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6258875Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6259183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6259282Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6259288Z 2025-12-04T09:46:16.6259407Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6259625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6259704Z res = mod(**inputs) 2025-12-04T09:46:16.6260009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6260086Z outputs = self.mobilebert( 2025-12-04T09:46:16.6260403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6260483Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6260796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6260874Z layer_outputs = layer_module( 2025-12-04T09:46:16.6261182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6261326Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6261633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6261726Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6261730Z 2025-12-04T09:46:16.6261850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6262088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6262187Z res = mod(**inputs) 2025-12-04T09:46:16.6262497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6262573Z outputs = self.mobilebert( 2025-12-04T09:46:16.6262910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6262989Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6263309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6263385Z layer_outputs = layer_module( 2025-12-04T09:46:16.6263700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6263842Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6264172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6264295Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6264307Z 2025-12-04T09:46:16.6264424Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6264641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6264718Z res = mod(**inputs) 2025-12-04T09:46:16.6265034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6265110Z outputs = self.mobilebert( 2025-12-04T09:46:16.6265424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6265503Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6265821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6265900Z layer_outputs = layer_module( 2025-12-04T09:46:16.6266211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6266393Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6266713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6266817Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6266830Z 2025-12-04T09:46:16.6266943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6267159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6267239Z res = mod(**inputs) 2025-12-04T09:46:16.6267545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6267623Z outputs = self.mobilebert( 2025-12-04T09:46:16.6267952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6268032Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6268347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6268423Z layer_outputs = layer_module( 2025-12-04T09:46:16.6268740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6268922Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6269253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6269411Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6269758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6269861Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6269865Z 2025-12-04T09:46:16.6269984Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6270200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6270270Z res = mod(**inputs) 2025-12-04T09:46:16.6270592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6270667Z outputs = self.mobilebert( 2025-12-04T09:46:16.6271007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6271086Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6271396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6271479Z layer_outputs = layer_module( 2025-12-04T09:46:16.6271798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6271976Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6272295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6272430Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6272752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6272846Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6272850Z 2025-12-04T09:46:16.6272975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6273194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6273265Z res = mod(**inputs) 2025-12-04T09:46:16.6273585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6273661Z outputs = self.mobilebert( 2025-12-04T09:46:16.6273969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6274055Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6274365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6274453Z layer_outputs = layer_module( 2025-12-04T09:46:16.6274760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6274930Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6275248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6275381Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6275698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6275832Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6276163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6276295Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6276299Z 2025-12-04T09:46:16.6276412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6276654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6276733Z res = mod(**inputs) 2025-12-04T09:46:16.6277043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6277129Z outputs = self.mobilebert( 2025-12-04T09:46:16.6277439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6277517Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6277834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6277931Z layer_outputs = layer_module( 2025-12-04T09:46:16.6278244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6278423Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6278732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6278863Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6279175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6279276Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6279280Z 2025-12-04T09:46:16.6279395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6279616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6279692Z res = mod(**inputs) 2025-12-04T09:46:16.6280000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6280079Z outputs = self.mobilebert( 2025-12-04T09:46:16.6280647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6280734Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6281050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6281129Z layer_outputs = layer_module( 2025-12-04T09:46:16.6281443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6281553Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6281865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6281956Z self_outputs = self.self( 2025-12-04T09:46:16.6282265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6282346Z self.value(value_tensor) 2025-12-04T09:46:16.6282350Z 2025-12-04T09:46:16.6282476Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6282693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6282764Z res = mod(**inputs) 2025-12-04T09:46:16.6283082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6283236Z outputs = self.mobilebert( 2025-12-04T09:46:16.6283585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6283663Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6283998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6284086Z layer_outputs = layer_module( 2025-12-04T09:46:16.6284403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6284591Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6284910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6285038Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6285379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6285471Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6285475Z 2025-12-04T09:46:16.6285590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6285818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6285891Z res = mod(**inputs) 2025-12-04T09:46:16.6286214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6286291Z outputs = self.mobilebert( 2025-12-04T09:46:16.6286607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6286695Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6287004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6287090Z layer_outputs = layer_module( 2025-12-04T09:46:16.6287398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6287575Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6287897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6288016Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6288335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6288439Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6288748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6288860Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6288864Z 2025-12-04T09:46:16.6288977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6289195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6289275Z res = mod(**inputs) 2025-12-04T09:46:16.6289596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6289684Z outputs = self.mobilebert( 2025-12-04T09:46:16.6290001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6290080Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6290415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6290509Z layer_outputs = layer_module( 2025-12-04T09:46:16.6290827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6290948Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6291258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6291345Z self_outputs = self.self( 2025-12-04T09:46:16.6291665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6291744Z self.query(query_tensor) 2025-12-04T09:46:16.6291748Z 2025-12-04T09:46:16.6291870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6292116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6292195Z res = mod(**inputs) 2025-12-04T09:46:16.6292505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6292582Z outputs = self.mobilebert( 2025-12-04T09:46:16.6292902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6292980Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6293300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6293382Z layer_outputs = layer_module( 2025-12-04T09:46:16.6293703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6293807Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6294119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6294196Z self_outputs = self.self( 2025-12-04T09:46:16.6294518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6294593Z self.key(key_tensor) 2025-12-04T09:46:16.6294597Z 2025-12-04T09:46:16.6294694Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6294782Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6294895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6295123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6295194Z res = mod(**inputs) 2025-12-04T09:46:16.6295505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6295592Z outputs = self.mobilebert( 2025-12-04T09:46:16.6295905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6295993Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6296305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6296383Z layer_outputs = layer_module( 2025-12-04T09:46:16.6296700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6296794Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6297111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6297268Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6297594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6297694Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6297698Z 2025-12-04T09:46:16.6297827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6298045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6298124Z res = mod(**inputs) 2025-12-04T09:46:16.6298429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6298516Z outputs = self.mobilebert( 2025-12-04T09:46:16.6298821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6298923Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6299244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6299324Z layer_outputs = layer_module( 2025-12-04T09:46:16.6299643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6299740Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6300050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6300197Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6300510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6300652Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6300973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6301078Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6301082Z 2025-12-04T09:46:16.6301208Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6301428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6301503Z res = mod(**inputs) 2025-12-04T09:46:16.6301824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6301903Z outputs = self.mobilebert( 2025-12-04T09:46:16.6302218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6302300Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6302610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6302704Z layer_outputs = layer_module( 2025-12-04T09:46:16.6303015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6303124Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6303444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6303572Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6303891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6303986Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6303990Z 2025-12-04T09:46:16.6304126Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6304376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6304448Z res = mod(**inputs) 2025-12-04T09:46:16.6304787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6304867Z outputs = self.mobilebert( 2025-12-04T09:46:16.6305188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6305276Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6305599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6305676Z layer_outputs = layer_module( 2025-12-04T09:46:16.6306060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6306191Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6306507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6306631Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6306946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6307077Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6307081Z 2025-12-04T09:46:16.6307192Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6307416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6307486Z res = mod(**inputs) 2025-12-04T09:46:16.6307818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6307903Z outputs = self.mobilebert( 2025-12-04T09:46:16.6308204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6308290Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6308607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6308685Z layer_outputs = layer_module( 2025-12-04T09:46:16.6309003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6309108Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6309425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6309573Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6309880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6309978Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6309982Z 2025-12-04T09:46:16.6310098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6310322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6310398Z res = mod(**inputs) 2025-12-04T09:46:16.6310706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6310786Z outputs = self.mobilebert( 2025-12-04T09:46:16.6311128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6311206Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6311533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6311609Z layer_outputs = layer_module( 2025-12-04T09:46:16.6311943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6312057Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6312363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6312506Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6312824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6312978Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6313304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6313404Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6313408Z 2025-12-04T09:46:16.6313530Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6313748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6313818Z res = mod(**inputs) 2025-12-04T09:46:16.6314142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6314219Z outputs = self.mobilebert( 2025-12-04T09:46:16.6314547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6314628Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6314942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6315027Z layer_outputs = layer_module( 2025-12-04T09:46:16.6315339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6315441Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6315763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6315886Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6316208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6316302Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6316309Z 2025-12-04T09:46:16.6316422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6316653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6316723Z res = mod(**inputs) 2025-12-04T09:46:16.6317046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6317125Z outputs = self.mobilebert( 2025-12-04T09:46:16.6317436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6317523Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6317835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6317913Z layer_outputs = layer_module( 2025-12-04T09:46:16.6318253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6318383Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6318701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6318838Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6319151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6319281Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6319285Z 2025-12-04T09:46:16.6319397Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6319621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6319691Z res = mod(**inputs) 2025-12-04T09:46:16.6320025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6320112Z outputs = self.mobilebert( 2025-12-04T09:46:16.6320503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6320593Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6320914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6320990Z layer_outputs = layer_module( 2025-12-04T09:46:16.6321305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6321409Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6321721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6321873Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6322190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6322289Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6322294Z 2025-12-04T09:46:16.6322404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6322615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6322692Z res = mod(**inputs) 2025-12-04T09:46:16.6322992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6323068Z outputs = self.mobilebert( 2025-12-04T09:46:16.6323378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6323462Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6323776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6323853Z layer_outputs = layer_module( 2025-12-04T09:46:16.6324163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6324276Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6324580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6324723Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6325038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6325198Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6325532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6325631Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6325635Z 2025-12-04T09:46:16.6325773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6325993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6326063Z res = mod(**inputs) 2025-12-04T09:46:16.6326386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6326462Z outputs = self.mobilebert( 2025-12-04T09:46:16.6326782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6326888Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6327195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6327278Z layer_outputs = layer_module( 2025-12-04T09:46:16.6327597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6327700Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6328014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6328136Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6328459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6328552Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6328557Z 2025-12-04T09:46:16.6328670Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6328893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6328962Z res = mod(**inputs) 2025-12-04T09:46:16.6329280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6329363Z outputs = self.mobilebert( 2025-12-04T09:46:16.6329668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6329754Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6330060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6330136Z layer_outputs = layer_module( 2025-12-04T09:46:16.6330451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6330554Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6330868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6330990Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6331306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6331434Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6331438Z 2025-12-04T09:46:16.6331550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6331773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6331842Z res = mod(**inputs) 2025-12-04T09:46:16.6332174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6332276Z outputs = self.mobilebert( 2025-12-04T09:46:16.6332610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6332691Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6333015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6333092Z layer_outputs = layer_module( 2025-12-04T09:46:16.6333418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6333519Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6333838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6334014Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6334323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6334426Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6334429Z 2025-12-04T09:46:16.6334543Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6334761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6334837Z res = mod(**inputs) 2025-12-04T09:46:16.6335146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6335226Z outputs = self.mobilebert( 2025-12-04T09:46:16.6335544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6335628Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6335942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6336019Z layer_outputs = layer_module( 2025-12-04T09:46:16.6336327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6336436Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6336746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6336891Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6337202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6337340Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6337654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6337755Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6337760Z 2025-12-04T09:46:16.6337882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6338099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6338169Z res = mod(**inputs) 2025-12-04T09:46:16.6338481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6338556Z outputs = self.mobilebert( 2025-12-04T09:46:16.6338887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6338992Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6339303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6339386Z layer_outputs = layer_module( 2025-12-04T09:46:16.6339716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6339850Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6340165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6340258Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6340261Z 2025-12-04T09:46:16.6340379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6340596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6340689Z res = mod(**inputs) 2025-12-04T09:46:16.6341004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6341084Z outputs = self.mobilebert( 2025-12-04T09:46:16.6341395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6341481Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6341792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6341876Z layer_outputs = layer_module( 2025-12-04T09:46:16.6342185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6342319Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6342637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6342760Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6342764Z 2025-12-04T09:46:16.6342884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6343105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6343175Z res = mod(**inputs) 2025-12-04T09:46:16.6343489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6343566Z outputs = self.mobilebert( 2025-12-04T09:46:16.6343876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6343965Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6344277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6344362Z layer_outputs = layer_module( 2025-12-04T09:46:16.6344674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6344851Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6345168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6345272Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6345276Z 2025-12-04T09:46:16.6345393Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6345611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6345705Z res = mod(**inputs) 2025-12-04T09:46:16.6346039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6346116Z outputs = self.mobilebert( 2025-12-04T09:46:16.6346454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6346542Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6346859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6346940Z layer_outputs = layer_module( 2025-12-04T09:46:16.6347448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6347627Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6348000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6348141Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6348456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6348558Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6348562Z 2025-12-04T09:46:16.6348674Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6348897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6348967Z res = mod(**inputs) 2025-12-04T09:46:16.6349288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6349366Z outputs = self.mobilebert( 2025-12-04T09:46:16.6349676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6349766Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6350074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6350152Z layer_outputs = layer_module( 2025-12-04T09:46:16.6350479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6350651Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6350966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6351104Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6351417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6351525Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6351528Z 2025-12-04T09:46:16.6351641Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6351868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6351936Z res = mod(**inputs) 2025-12-04T09:46:16.6352241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6352328Z outputs = self.mobilebert( 2025-12-04T09:46:16.6352648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6352727Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6353082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6353186Z layer_outputs = layer_module( 2025-12-04T09:46:16.6353503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6353700Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6354020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6354162Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6354485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6354626Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6354948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6355071Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6355075Z 2025-12-04T09:46:16.6355197Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6355418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6355496Z res = mod(**inputs) 2025-12-04T09:46:16.6355812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6355891Z outputs = self.mobilebert( 2025-12-04T09:46:16.6356215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6356294Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6356602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6356690Z layer_outputs = layer_module( 2025-12-04T09:46:16.6356994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6357178Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6357484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6357608Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6357924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6358017Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6358020Z 2025-12-04T09:46:16.6358143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6358365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6358435Z res = mod(**inputs) 2025-12-04T09:46:16.6358748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6358828Z outputs = self.mobilebert( 2025-12-04T09:46:16.6359135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6359222Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6359524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6359607Z layer_outputs = layer_module( 2025-12-04T09:46:16.6359933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6360050Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6360417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6360504Z self_outputs = self.self( 2025-12-04T09:46:16.6360850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6360933Z self.value(value_tensor) 2025-12-04T09:46:16.6360937Z 2025-12-04T09:46:16.6361054Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6361276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6361346Z res = mod(**inputs) 2025-12-04T09:46:16.6361653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6361764Z outputs = self.mobilebert( 2025-12-04T09:46:16.6362075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6362162Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6362471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6362549Z layer_outputs = layer_module( 2025-12-04T09:46:16.6362865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6363040Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6363357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6363483Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6363797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6363900Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6363904Z 2025-12-04T09:46:16.6364017Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6364241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6364312Z res = mod(**inputs) 2025-12-04T09:46:16.6364618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6364702Z outputs = self.mobilebert( 2025-12-04T09:46:16.6365010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6365089Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6365405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6365482Z layer_outputs = layer_module( 2025-12-04T09:46:16.6365799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6365974Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6366282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6366413Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6366730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6366833Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6367163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6367282Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6367286Z 2025-12-04T09:46:16.6367407Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6367645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6367719Z res = mod(**inputs) 2025-12-04T09:46:16.6368037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6368113Z outputs = self.mobilebert( 2025-12-04T09:46:16.6368435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6368514Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6368847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6368934Z layer_outputs = layer_module( 2025-12-04T09:46:16.6369252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6369355Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6369684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6369761Z self_outputs = self.self( 2025-12-04T09:46:16.6370095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6370175Z self.query(query_tensor) 2025-12-04T09:46:16.6370179Z 2025-12-04T09:46:16.6370295Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6370527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6370599Z res = mod(**inputs) 2025-12-04T09:46:16.6370921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6370998Z outputs = self.mobilebert( 2025-12-04T09:46:16.6371325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6371416Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6371745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6371829Z layer_outputs = layer_module( 2025-12-04T09:46:16.6372160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6372258Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6372585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6372663Z self_outputs = self.self( 2025-12-04T09:46:16.6372992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6373075Z self.key(key_tensor) 2025-12-04T09:46:16.6373079Z 2025-12-04T09:46:16.6373168Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6373264Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6373379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6373598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6373679Z res = mod(**inputs) 2025-12-04T09:46:16.6374027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6374122Z outputs = self.mobilebert( 2025-12-04T09:46:16.6374436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6374515Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6374855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6374935Z layer_outputs = layer_module( 2025-12-04T09:46:16.6375251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6375350Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6375665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6375832Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6376139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6376230Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6376234Z 2025-12-04T09:46:16.6376354Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6376570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6376647Z res = mod(**inputs) 2025-12-04T09:46:16.6376962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6377040Z outputs = self.mobilebert( 2025-12-04T09:46:16.6377353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6377434Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6377743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6377828Z layer_outputs = layer_module( 2025-12-04T09:46:16.6378135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6378234Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6378540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6378674Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6378990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6379128Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6379445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6379545Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6379549Z 2025-12-04T09:46:16.6379661Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6379884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6379953Z res = mod(**inputs) 2025-12-04T09:46:16.6380259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6380342Z outputs = self.mobilebert( 2025-12-04T09:46:16.6380646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6380732Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6381060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6381156Z layer_outputs = layer_module( 2025-12-04T09:46:16.6381476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6381598Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6381919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6382042Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6382350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6382450Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6382454Z 2025-12-04T09:46:16.6382588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6382815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6382886Z res = mod(**inputs) 2025-12-04T09:46:16.6383194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6383277Z outputs = self.mobilebert( 2025-12-04T09:46:16.6383584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6383661Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6383976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6384052Z layer_outputs = layer_module( 2025-12-04T09:46:16.6384367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6384474Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6384785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6384916Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6385224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6385356Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6385360Z 2025-12-04T09:46:16.6385474Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6385695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6385772Z res = mod(**inputs) 2025-12-04T09:46:16.6386079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6386160Z outputs = self.mobilebert( 2025-12-04T09:46:16.6386475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6386555Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6386871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6386948Z layer_outputs = layer_module( 2025-12-04T09:46:16.6387255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6387368Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6387686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6387857Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6388191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6388284Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6388288Z 2025-12-04T09:46:16.6388430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6388650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6388722Z res = mod(**inputs) 2025-12-04T09:46:16.6389045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6389123Z outputs = self.mobilebert( 2025-12-04T09:46:16.6389447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6389588Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6389906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6389992Z layer_outputs = layer_module( 2025-12-04T09:46:16.6390322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6390434Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6390751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6390888Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6391219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6391357Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6391678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6391786Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6391790Z 2025-12-04T09:46:16.6391903Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6392134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6392204Z res = mod(**inputs) 2025-12-04T09:46:16.6392522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6392608Z outputs = self.mobilebert( 2025-12-04T09:46:16.6392923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6393013Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6393329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6393406Z layer_outputs = layer_module( 2025-12-04T09:46:16.6393736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6393842Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6394169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6394293Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6394612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6394713Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6394719Z 2025-12-04T09:46:16.6394856Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6395095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6395172Z res = mod(**inputs) 2025-12-04T09:46:16.6395500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6395585Z outputs = self.mobilebert( 2025-12-04T09:46:16.6395893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6395972Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6396286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6396363Z layer_outputs = layer_module( 2025-12-04T09:46:16.6396686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6396811Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6397122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6397252Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6397561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6397684Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6397696Z 2025-12-04T09:46:16.6397810Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6398028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6398106Z res = mod(**inputs) 2025-12-04T09:46:16.6398418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6398495Z outputs = self.mobilebert( 2025-12-04T09:46:16.6398817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6398897Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6399217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6399294Z layer_outputs = layer_module( 2025-12-04T09:46:16.6399604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6399715Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6400025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6400165Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6400573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6400672Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6400678Z 2025-12-04T09:46:16.6400797Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6401012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6401083Z res = mod(**inputs) 2025-12-04T09:46:16.6401402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6401480Z outputs = self.mobilebert( 2025-12-04T09:46:16.6401827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6401922Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6402219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6402303Z layer_outputs = layer_module( 2025-12-04T09:46:16.6402619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6402721Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6403036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6403174Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6403494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6403653Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6403961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6404069Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6404072Z 2025-12-04T09:46:16.6404189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6404416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6404486Z res = mod(**inputs) 2025-12-04T09:46:16.6404790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6404874Z outputs = self.mobilebert( 2025-12-04T09:46:16.6405180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6405270Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6405577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6405652Z layer_outputs = layer_module( 2025-12-04T09:46:16.6405966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6406078Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6406375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6406504Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6406810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6406913Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6406918Z 2025-12-04T09:46:16.6407032Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6407248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6407328Z res = mod(**inputs) 2025-12-04T09:46:16.6407645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6407731Z outputs = self.mobilebert( 2025-12-04T09:46:16.6408043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6408121Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6408439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6408515Z layer_outputs = layer_module( 2025-12-04T09:46:16.6408844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6408977Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6409283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6409430Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6409738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6409859Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6409863Z 2025-12-04T09:46:16.6409982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6410197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6410273Z res = mod(**inputs) 2025-12-04T09:46:16.6410595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6410673Z outputs = self.mobilebert( 2025-12-04T09:46:16.6410988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6411068Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6411384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6411467Z layer_outputs = layer_module( 2025-12-04T09:46:16.6411779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6411887Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6412255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6412393Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6412709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6412802Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6412805Z 2025-12-04T09:46:16.6412924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6413139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6413210Z res = mod(**inputs) 2025-12-04T09:46:16.6413523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6413598Z outputs = self.mobilebert( 2025-12-04T09:46:16.6413919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6414001Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6414305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6414388Z layer_outputs = layer_module( 2025-12-04T09:46:16.6414706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6414807Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6415132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6415267Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6415623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6415779Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6416089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6416197Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6416219Z 2025-12-04T09:46:16.6416334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6416557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6416626Z res = mod(**inputs) 2025-12-04T09:46:16.6416942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6417026Z outputs = self.mobilebert( 2025-12-04T09:46:16.6417345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6417444Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6417765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6417842Z layer_outputs = layer_module( 2025-12-04T09:46:16.6418165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6418299Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6418616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6418717Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6418721Z 2025-12-04T09:46:16.6418833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6419064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6419136Z res = mod(**inputs) 2025-12-04T09:46:16.6419452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6419538Z outputs = self.mobilebert( 2025-12-04T09:46:16.6419852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6419931Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6420252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6420328Z layer_outputs = layer_module( 2025-12-04T09:46:16.6420649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6420783Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6421099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6421231Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6421235Z 2025-12-04T09:46:16.6421346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6421571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6421640Z res = mod(**inputs) 2025-12-04T09:46:16.6421951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6422038Z outputs = self.mobilebert( 2025-12-04T09:46:16.6422351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6422436Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6422777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6422880Z layer_outputs = layer_module( 2025-12-04T09:46:16.6423245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6423422Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6423728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6423841Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6423845Z 2025-12-04T09:46:16.6423957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6424185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6424277Z res = mod(**inputs) 2025-12-04T09:46:16.6424589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6424674Z outputs = self.mobilebert( 2025-12-04T09:46:16.6424983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6425071Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6425381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6425458Z layer_outputs = layer_module( 2025-12-04T09:46:16.6425776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6425952Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6426265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6426409Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6426721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6426832Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6426836Z 2025-12-04T09:46:16.6426950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6427167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6427245Z res = mod(**inputs) 2025-12-04T09:46:16.6427556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6427639Z outputs = self.mobilebert( 2025-12-04T09:46:16.6427951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6428031Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6428356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6428434Z layer_outputs = layer_module( 2025-12-04T09:46:16.6428760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6428932Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6429251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6429394Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6429727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6429844Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6429861Z 2025-12-04T09:46:16.6429973Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6430204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6430284Z res = mod(**inputs) 2025-12-04T09:46:16.6430601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6430677Z outputs = self.mobilebert( 2025-12-04T09:46:16.6430994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6431073Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6431400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6431501Z layer_outputs = layer_module( 2025-12-04T09:46:16.6431809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6431988Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6432309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6432453Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6432772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6432908Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6433232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6433338Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6433342Z 2025-12-04T09:46:16.6433454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6433679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6433748Z res = mod(**inputs) 2025-12-04T09:46:16.6434063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6434140Z outputs = self.mobilebert( 2025-12-04T09:46:16.6434457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6434546Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6434867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6434954Z layer_outputs = layer_module( 2025-12-04T09:46:16.6435262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6435439Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6435757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6435878Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6436197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6436294Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6436298Z 2025-12-04T09:46:16.6436438Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6436666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6436757Z res = mod(**inputs) 2025-12-04T09:46:16.6437065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6437168Z outputs = self.mobilebert( 2025-12-04T09:46:16.6437476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6437560Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6437886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6437961Z layer_outputs = layer_module( 2025-12-04T09:46:16.6438276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6438409Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6438718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6438802Z self_outputs = self.self( 2025-12-04T09:46:16.6439111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6439197Z self.value(value_tensor) 2025-12-04T09:46:16.6439201Z 2025-12-04T09:46:16.6439313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6439527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6439607Z res = mod(**inputs) 2025-12-04T09:46:16.6439907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6439996Z outputs = self.mobilebert( 2025-12-04T09:46:16.6440307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6440470Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6440801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6440882Z layer_outputs = layer_module( 2025-12-04T09:46:16.6441190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6441378Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6441689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6441823Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6442138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6442229Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6442233Z 2025-12-04T09:46:16.6442357Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6442577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6442655Z res = mod(**inputs) 2025-12-04T09:46:16.6442964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6443042Z outputs = self.mobilebert( 2025-12-04T09:46:16.6443359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6443441Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6443789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6443884Z layer_outputs = layer_module( 2025-12-04T09:46:16.6444222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6444407Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6444717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6444838Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6445151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6445247Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6445585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6445687Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6445692Z 2025-12-04T09:46:16.6445806Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6446033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6446101Z res = mod(**inputs) 2025-12-04T09:46:16.6446414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6446492Z outputs = self.mobilebert( 2025-12-04T09:46:16.6446796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6446882Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6447387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6447474Z layer_outputs = layer_module( 2025-12-04T09:46:16.6447795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6447893Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6448211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6448291Z self_outputs = self.self( 2025-12-04T09:46:16.6448625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6448712Z self.query(query_tensor) 2025-12-04T09:46:16.6448717Z 2025-12-04T09:46:16.6448831Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6449068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6449143Z res = mod(**inputs) 2025-12-04T09:46:16.6449453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6449538Z outputs = self.mobilebert( 2025-12-04T09:46:16.6449872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6449951Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6450269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6450346Z layer_outputs = layer_module( 2025-12-04T09:46:16.6450681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6450825Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6451156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6451241Z self_outputs = self.self( 2025-12-04T09:46:16.6451577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6451660Z self.key(key_tensor) 2025-12-04T09:46:16.6451664Z 2025-12-04T09:46:16.6451752Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6451840Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6451960Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6452176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6452248Z res = mod(**inputs) 2025-12-04T09:46:16.6452588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6452698Z outputs = self.mobilebert( 2025-12-04T09:46:16.6453023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6453102Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6453432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6453520Z layer_outputs = layer_module( 2025-12-04T09:46:16.6453854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6453950Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6454296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6454436Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6454766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6454860Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6454863Z 2025-12-04T09:46:16.6454976Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6455207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6455278Z res = mod(**inputs) 2025-12-04T09:46:16.6455613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6455690Z outputs = self.mobilebert( 2025-12-04T09:46:16.6456021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6456112Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6456435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6456520Z layer_outputs = layer_module( 2025-12-04T09:46:16.6456852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6456946Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6457273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6457408Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6457737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6457882Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6458239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6458366Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6458370Z 2025-12-04T09:46:16.6458481Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6458717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6458798Z res = mod(**inputs) 2025-12-04T09:46:16.6459105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6459189Z outputs = self.mobilebert( 2025-12-04T09:46:16.6459495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6459575Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6459912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6459988Z layer_outputs = layer_module( 2025-12-04T09:46:16.6460299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6460422Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6460724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6460851Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6461151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6461243Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6461247Z 2025-12-04T09:46:16.6461373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6461593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6461671Z res = mod(**inputs) 2025-12-04T09:46:16.6461981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6462058Z outputs = self.mobilebert( 2025-12-04T09:46:16.6462376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6462455Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6462763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6462849Z layer_outputs = layer_module( 2025-12-04T09:46:16.6463155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6463269Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6463577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6463701Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6464015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6464140Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6464144Z 2025-12-04T09:46:16.6464263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6464480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6464550Z res = mod(**inputs) 2025-12-04T09:46:16.6464904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6465003Z outputs = self.mobilebert( 2025-12-04T09:46:16.6465320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6465398Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6465731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6465817Z layer_outputs = layer_module( 2025-12-04T09:46:16.6466124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6466230Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6466545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6466707Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6467021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6467115Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6467119Z 2025-12-04T09:46:16.6467233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6467459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6467529Z res = mod(**inputs) 2025-12-04T09:46:16.6467841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6467917Z outputs = self.mobilebert( 2025-12-04T09:46:16.6468223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6468312Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6468618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6468695Z layer_outputs = layer_module( 2025-12-04T09:46:16.6469011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6469115Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6469430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6469569Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6469876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6470022Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6470334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6470440Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6470443Z 2025-12-04T09:46:16.6470557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6470773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6470853Z res = mod(**inputs) 2025-12-04T09:46:16.6471169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6471247Z outputs = self.mobilebert( 2025-12-04T09:46:16.6471570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6471672Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6471987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6472082Z layer_outputs = layer_module( 2025-12-04T09:46:16.6472419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6472528Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6472835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6472961Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6473280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6473370Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6473400Z 2025-12-04T09:46:16.6473523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6473741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6473818Z res = mod(**inputs) 2025-12-04T09:46:16.6474136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6474212Z outputs = self.mobilebert( 2025-12-04T09:46:16.6474531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6474610Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6474928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6475013Z layer_outputs = layer_module( 2025-12-04T09:46:16.6475339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6475448Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6475748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6475866Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6476172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6476292Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6476295Z 2025-12-04T09:46:16.6476416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6476638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6476708Z res = mod(**inputs) 2025-12-04T09:46:16.6477028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6477108Z outputs = self.mobilebert( 2025-12-04T09:46:16.6477415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6477503Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6477824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6477909Z layer_outputs = layer_module( 2025-12-04T09:46:16.6478229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6478333Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6478667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6478826Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6479159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6479253Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6479272Z 2025-12-04T09:46:16.6479388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6479613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6479681Z res = mod(**inputs) 2025-12-04T09:46:16.6480003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6480089Z outputs = self.mobilebert( 2025-12-04T09:46:16.6480476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6480594Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6480903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6480980Z layer_outputs = layer_module( 2025-12-04T09:46:16.6481303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6481409Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6481730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6481870Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6482181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6482326Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6482640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6482748Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6482752Z 2025-12-04T09:46:16.6482868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6483084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6483166Z res = mod(**inputs) 2025-12-04T09:46:16.6483477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6483555Z outputs = self.mobilebert( 2025-12-04T09:46:16.6483876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6483960Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6484282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6484358Z layer_outputs = layer_module( 2025-12-04T09:46:16.6484672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6488253Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6488591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6488715Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6489037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6489131Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6489140Z 2025-12-04T09:46:16.6489293Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6489515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6489586Z res = mod(**inputs) 2025-12-04T09:46:16.6489925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6490005Z outputs = self.mobilebert( 2025-12-04T09:46:16.6490357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6490437Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6490755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6490832Z layer_outputs = layer_module( 2025-12-04T09:46:16.6491168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6491281Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6491592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6491722Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6492108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6492233Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6492237Z 2025-12-04T09:46:16.6492355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6492577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6492654Z res = mod(**inputs) 2025-12-04T09:46:16.6492966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6493047Z outputs = self.mobilebert( 2025-12-04T09:46:16.6493364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6493445Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6493759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6493843Z layer_outputs = layer_module( 2025-12-04T09:46:16.6494158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6494266Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6494581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6494718Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6495026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6495118Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6495121Z 2025-12-04T09:46:16.6495238Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6495525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6495596Z res = mod(**inputs) 2025-12-04T09:46:16.6495902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6495978Z outputs = self.mobilebert( 2025-12-04T09:46:16.6496284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6496402Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6496702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6496785Z layer_outputs = layer_module( 2025-12-04T09:46:16.6497113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6497216Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6497519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6497650Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6497963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6498117Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6498413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6498517Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6498522Z 2025-12-04T09:46:16.6498631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6498849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6498916Z res = mod(**inputs) 2025-12-04T09:46:16.6499209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6499293Z outputs = self.mobilebert( 2025-12-04T09:46:16.6499587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6499666Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6499967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6500041Z layer_outputs = layer_module( 2025-12-04T09:46:16.6500362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6500496Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6500802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6500903Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6500906Z 2025-12-04T09:46:16.6501016Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6501235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6501308Z res = mod(**inputs) 2025-12-04T09:46:16.6501618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6501702Z outputs = self.mobilebert( 2025-12-04T09:46:16.6502018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6502139Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6502446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6502520Z layer_outputs = layer_module( 2025-12-04T09:46:16.6502824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6502954Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6503250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6503399Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6503402Z 2025-12-04T09:46:16.6503511Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6503782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6503851Z res = mod(**inputs) 2025-12-04T09:46:16.6504171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6504245Z outputs = self.mobilebert( 2025-12-04T09:46:16.6504541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6504624Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6504941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6505024Z layer_outputs = layer_module( 2025-12-04T09:46:16.6505326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6505496Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6505803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6505904Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6505908Z 2025-12-04T09:46:16.6506024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6506233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6506301Z res = mod(**inputs) 2025-12-04T09:46:16.6506609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6506684Z outputs = self.mobilebert( 2025-12-04T09:46:16.6506990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6507065Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6507367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6507450Z layer_outputs = layer_module( 2025-12-04T09:46:16.6507749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6507918Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6508224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6508359Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6508668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6508766Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6508770Z 2025-12-04T09:46:16.6508907Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6509128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6509198Z res = mod(**inputs) 2025-12-04T09:46:16.6509501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6509575Z outputs = self.mobilebert( 2025-12-04T09:46:16.6509878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6509985Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6510289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6510380Z layer_outputs = layer_module( 2025-12-04T09:46:16.6510691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6510854Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6511159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6511290Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6511588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6511706Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6511709Z 2025-12-04T09:46:16.6511820Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6512036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6512103Z res = mod(**inputs) 2025-12-04T09:46:16.6512402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6512483Z outputs = self.mobilebert( 2025-12-04T09:46:16.6512781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6512866Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6513162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6513238Z layer_outputs = layer_module( 2025-12-04T09:46:16.6513541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6513705Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6514003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6514142Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6514436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6514574Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6514872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6514988Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6514992Z 2025-12-04T09:46:16.6515108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6515320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6515396Z res = mod(**inputs) 2025-12-04T09:46:16.6515716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6515794Z outputs = self.mobilebert( 2025-12-04T09:46:16.6516098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6516175Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6516480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6516578Z layer_outputs = layer_module( 2025-12-04T09:46:16.6516887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6517088Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6517399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6517521Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6517837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6517926Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6517930Z 2025-12-04T09:46:16.6518051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6518287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6518359Z res = mod(**inputs) 2025-12-04T09:46:16.6518674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6518755Z outputs = self.mobilebert( 2025-12-04T09:46:16.6519070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6519149Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6519457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6519542Z layer_outputs = layer_module( 2025-12-04T09:46:16.6519851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6519950Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6520268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6520430Z self_outputs = self.self( 2025-12-04T09:46:16.6520770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6520854Z self.value(value_tensor) 2025-12-04T09:46:16.6520858Z 2025-12-04T09:46:16.6520971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6521200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6521271Z res = mod(**inputs) 2025-12-04T09:46:16.6521592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6521671Z outputs = self.mobilebert( 2025-12-04T09:46:16.6521990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6522079Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6522393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6522472Z layer_outputs = layer_module( 2025-12-04T09:46:16.6522833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6523012Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6523329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6523451Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6523759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6523880Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6523884Z 2025-12-04T09:46:16.6523998Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6524244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6524319Z res = mod(**inputs) 2025-12-04T09:46:16.6524628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6524714Z outputs = self.mobilebert( 2025-12-04T09:46:16.6525022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6525100Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6525436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6525516Z layer_outputs = layer_module( 2025-12-04T09:46:16.6525830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6526006Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6526319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6526451Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6526757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6526857Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6527168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6527269Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6527273Z 2025-12-04T09:46:16.6527392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6527611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6527688Z res = mod(**inputs) 2025-12-04T09:46:16.6527995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6528072Z outputs = self.mobilebert( 2025-12-04T09:46:16.6528383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6528462Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6528766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6528852Z layer_outputs = layer_module( 2025-12-04T09:46:16.6529157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6529259Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6529586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6529666Z self_outputs = self.self( 2025-12-04T09:46:16.6529980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6530058Z self.query(query_tensor) 2025-12-04T09:46:16.6530062Z 2025-12-04T09:46:16.6530180Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6530393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6530486Z res = mod(**inputs) 2025-12-04T09:46:16.6530805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6530880Z outputs = self.mobilebert( 2025-12-04T09:46:16.6531207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6531299Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6531606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6531688Z layer_outputs = layer_module( 2025-12-04T09:46:16.6531992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6532104Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6532420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6532496Z self_outputs = self.self( 2025-12-04T09:46:16.6532830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6532904Z self.key(key_tensor) 2025-12-04T09:46:16.6532908Z 2025-12-04T09:46:16.6532998Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6533092Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6533204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6533418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6533494Z res = mod(**inputs) 2025-12-04T09:46:16.6533820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6533907Z outputs = self.mobilebert( 2025-12-04T09:46:16.6534212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6534290Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6534621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6534701Z layer_outputs = layer_module( 2025-12-04T09:46:16.6535007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6535109Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6535439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6535596Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6535895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6535985Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6535988Z 2025-12-04T09:46:16.6536107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6536342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6536421Z res = mod(**inputs) 2025-12-04T09:46:16.6536719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6536796Z outputs = self.mobilebert( 2025-12-04T09:46:16.6537106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6537185Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6537508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6537594Z layer_outputs = layer_module( 2025-12-04T09:46:16.6537924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6538027Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6538353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6538489Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6538822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6538956Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6539287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6539388Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6539392Z 2025-12-04T09:46:16.6539500Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6539719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6539789Z res = mod(**inputs) 2025-12-04T09:46:16.6540094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6540168Z outputs = self.mobilebert( 2025-12-04T09:46:16.6540468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6540556Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6540865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6540943Z layer_outputs = layer_module( 2025-12-04T09:46:16.6541258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6541364Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6541681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6541806Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6542121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6542223Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6542226Z 2025-12-04T09:46:16.6542339Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6542575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6542643Z res = mod(**inputs) 2025-12-04T09:46:16.6542944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6543028Z outputs = self.mobilebert( 2025-12-04T09:46:16.6543355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6543437Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6543748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6543824Z layer_outputs = layer_module( 2025-12-04T09:46:16.6544138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6544284Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6544590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6544718Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6545051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6545184Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6545187Z 2025-12-04T09:46:16.6545296Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6545506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6545583Z res = mod(**inputs) 2025-12-04T09:46:16.6545880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6545985Z outputs = self.mobilebert( 2025-12-04T09:46:16.6546287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6546363Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6546667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6546742Z layer_outputs = layer_module( 2025-12-04T09:46:16.6547333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6547455Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6547758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6547906Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6548209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6548299Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6548303Z 2025-12-04T09:46:16.6548423Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6548633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6548708Z res = mod(**inputs) 2025-12-04T09:46:16.6549004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6549080Z outputs = self.mobilebert( 2025-12-04T09:46:16.6549385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6549464Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6549765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6549848Z layer_outputs = layer_module( 2025-12-04T09:46:16.6550147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6550307Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6550607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6550742Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6551050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6551181Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6551515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6551613Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6551617Z 2025-12-04T09:46:16.6551750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6551975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6552048Z res = mod(**inputs) 2025-12-04T09:46:16.6552352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6552439Z outputs = self.mobilebert( 2025-12-04T09:46:16.6552747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6552835Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6553172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6553262Z layer_outputs = layer_module( 2025-12-04T09:46:16.6553570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6553672Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6553975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6554093Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6554391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6554488Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6554493Z 2025-12-04T09:46:16.6554602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6554845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6554914Z res = mod(**inputs) 2025-12-04T09:46:16.6555207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6555288Z outputs = self.mobilebert( 2025-12-04T09:46:16.6555587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6555663Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6555970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6556045Z layer_outputs = layer_module( 2025-12-04T09:46:16.6556350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6556453Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6556748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6556873Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6557190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6557323Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6557327Z 2025-12-04T09:46:16.6557437Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6557652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6557730Z res = mod(**inputs) 2025-12-04T09:46:16.6558039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6558139Z outputs = self.mobilebert( 2025-12-04T09:46:16.6558453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6558549Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6558876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6558951Z layer_outputs = layer_module( 2025-12-04T09:46:16.6559246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6559352Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6559658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6559830Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6560142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6560237Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6560243Z 2025-12-04T09:46:16.6560426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6560654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6560726Z res = mod(**inputs) 2025-12-04T09:46:16.6561044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6561120Z outputs = self.mobilebert( 2025-12-04T09:46:16.6561436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6561519Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6561830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6561916Z layer_outputs = layer_module( 2025-12-04T09:46:16.6562224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6562338Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6562648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6562784Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6563103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6563239Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6563547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6563658Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6563661Z 2025-12-04T09:46:16.6563776Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6564024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6564098Z res = mod(**inputs) 2025-12-04T09:46:16.6564407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6564495Z outputs = self.mobilebert( 2025-12-04T09:46:16.6564801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6564888Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6565209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6565285Z layer_outputs = layer_module( 2025-12-04T09:46:16.6565618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6565720Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6566040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6566162Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6566471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6566569Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6566591Z 2025-12-04T09:46:16.6566705Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6566923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6567002Z res = mod(**inputs) 2025-12-04T09:46:16.6567312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6567395Z outputs = self.mobilebert( 2025-12-04T09:46:16.6567708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6567785Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6568099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6568176Z layer_outputs = layer_module( 2025-12-04T09:46:16.6568491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6568595Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6568903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6569034Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6569345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6569469Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6569479Z 2025-12-04T09:46:16.6569592Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6569810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6569890Z res = mod(**inputs) 2025-12-04T09:46:16.6570200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6570279Z outputs = self.mobilebert( 2025-12-04T09:46:16.6570595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6570674Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6571012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6571093Z layer_outputs = layer_module( 2025-12-04T09:46:16.6571405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6571514Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6571811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6571970Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6572277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6572399Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6572403Z 2025-12-04T09:46:16.6572523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6572735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6572803Z res = mod(**inputs) 2025-12-04T09:46:16.6573114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6573189Z outputs = self.mobilebert( 2025-12-04T09:46:16.6573495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6573594Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6573891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6573972Z layer_outputs = layer_module( 2025-12-04T09:46:16.6574279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6574379Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6574694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6574826Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6575127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6575258Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6575559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6575673Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6575677Z 2025-12-04T09:46:16.6575788Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6576007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6576075Z res = mod(**inputs) 2025-12-04T09:46:16.6576382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6576465Z outputs = self.mobilebert( 2025-12-04T09:46:16.6576771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6576855Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6577154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6577228Z layer_outputs = layer_module( 2025-12-04T09:46:16.6577533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6577689Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6578022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6578116Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6578120Z 2025-12-04T09:46:16.6578232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6578456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6578551Z res = mod(**inputs) 2025-12-04T09:46:16.6578859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6578946Z outputs = self.mobilebert( 2025-12-04T09:46:16.6579285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6579375Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6579695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6579771Z layer_outputs = layer_module( 2025-12-04T09:46:16.6580085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6580225Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6580549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6580668Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6580673Z 2025-12-04T09:46:16.6580780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6580996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6581067Z res = mod(**inputs) 2025-12-04T09:46:16.6581363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6581446Z outputs = self.mobilebert( 2025-12-04T09:46:16.6581751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6581834Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6582138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6582217Z layer_outputs = layer_module( 2025-12-04T09:46:16.6582530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6582714Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6583023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6583125Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6583129Z 2025-12-04T09:46:16.6583238Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6583455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6583525Z res = mod(**inputs) 2025-12-04T09:46:16.6583821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6583905Z outputs = self.mobilebert( 2025-12-04T09:46:16.6584201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6584285Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6584600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6584678Z layer_outputs = layer_module( 2025-12-04T09:46:16.6584984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6585154Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6585458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6585614Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6585924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6586032Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6586036Z 2025-12-04T09:46:16.6586149Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6586372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6586443Z res = mod(**inputs) 2025-12-04T09:46:16.6586745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6586831Z outputs = self.mobilebert( 2025-12-04T09:46:16.6587151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6587228Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6587532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6587607Z layer_outputs = layer_module( 2025-12-04T09:46:16.6587913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6588079Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6588374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6588513Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6588809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6588909Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6588913Z 2025-12-04T09:46:16.6589024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6589236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6589312Z res = mod(**inputs) 2025-12-04T09:46:16.6589611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6589686Z outputs = self.mobilebert( 2025-12-04T09:46:16.6589989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6590066Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6590371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6590448Z layer_outputs = layer_module( 2025-12-04T09:46:16.6590749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6590924Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6591253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6591397Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6591694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6591824Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6592128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6592248Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6592252Z 2025-12-04T09:46:16.6592367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6592596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6592667Z res = mod(**inputs) 2025-12-04T09:46:16.6592970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6593044Z outputs = self.mobilebert( 2025-12-04T09:46:16.6593340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6593424Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6593719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6593821Z layer_outputs = layer_module( 2025-12-04T09:46:16.6594130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6594306Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6594626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6594745Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6595077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6595167Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6595172Z 2025-12-04T09:46:16.6595282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6595514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6595585Z res = mod(**inputs) 2025-12-04T09:46:16.6595900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6595986Z outputs = self.mobilebert( 2025-12-04T09:46:16.6596305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6596390Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6596708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6596783Z layer_outputs = layer_module( 2025-12-04T09:46:16.6597122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6597219Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6597554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6597629Z self_outputs = self.self( 2025-12-04T09:46:16.6597944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6598051Z self.value(value_tensor) 2025-12-04T09:46:16.6598056Z 2025-12-04T09:46:16.6598170Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6598395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6598466Z res = mod(**inputs) 2025-12-04T09:46:16.6598775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6598862Z outputs = self.mobilebert( 2025-12-04T09:46:16.6599204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6599283Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6599625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6599704Z layer_outputs = layer_module( 2025-12-04T09:46:16.6600029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6600203Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6600603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6600744Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6601080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6601179Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6601184Z 2025-12-04T09:46:16.6601298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6601517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6601598Z res = mod(**inputs) 2025-12-04T09:46:16.6601924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6602001Z outputs = self.mobilebert( 2025-12-04T09:46:16.6602341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6602421Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6602736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6602816Z layer_outputs = layer_module( 2025-12-04T09:46:16.6603123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6603308Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6603621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6603751Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6604094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6604188Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6604501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6604602Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6604606Z 2025-12-04T09:46:16.6604724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6604939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6605009Z res = mod(**inputs) 2025-12-04T09:46:16.6605368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6605444Z outputs = self.mobilebert( 2025-12-04T09:46:16.6605752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6605841Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6606151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6606259Z layer_outputs = layer_module( 2025-12-04T09:46:16.6606575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6606690Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6607012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6607091Z self_outputs = self.self( 2025-12-04T09:46:16.6607398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6607487Z self.query(query_tensor) 2025-12-04T09:46:16.6607491Z 2025-12-04T09:46:16.6607603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6607828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6607922Z res = mod(**inputs) 2025-12-04T09:46:16.6608233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6608320Z outputs = self.mobilebert( 2025-12-04T09:46:16.6608625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6608716Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6609025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6609103Z layer_outputs = layer_module( 2025-12-04T09:46:16.6609418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6609515Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6609825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6609910Z self_outputs = self.self( 2025-12-04T09:46:16.6610216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6610300Z self.key(key_tensor) 2025-12-04T09:46:16.6610304Z 2025-12-04T09:46:16.6610395Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6610482Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6610603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6610817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6610894Z res = mod(**inputs) 2025-12-04T09:46:16.6611200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6611279Z outputs = self.mobilebert( 2025-12-04T09:46:16.6611591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6611669Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6611978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6612081Z layer_outputs = layer_module( 2025-12-04T09:46:16.6612392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6612491Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6612799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6612935Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6613305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6613397Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6613400Z 2025-12-04T09:46:16.6613536Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6613756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6613827Z res = mod(**inputs) 2025-12-04T09:46:16.6614146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6614221Z outputs = self.mobilebert( 2025-12-04T09:46:16.6614523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6614607Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6614925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6615007Z layer_outputs = layer_module( 2025-12-04T09:46:16.6615307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6615399Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6615708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6615839Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6616153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6616286Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6616594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6616705Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6616708Z 2025-12-04T09:46:16.6616819Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6617035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6617116Z res = mod(**inputs) 2025-12-04T09:46:16.6617433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6617516Z outputs = self.mobilebert( 2025-12-04T09:46:16.6617839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6617917Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6618223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6618299Z layer_outputs = layer_module( 2025-12-04T09:46:16.6618607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6618708Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6619034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6619165Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6619462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6619552Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6619562Z 2025-12-04T09:46:16.6619672Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6619910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6619987Z res = mod(**inputs) 2025-12-04T09:46:16.6620303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6620397Z outputs = self.mobilebert( 2025-12-04T09:46:16.6620725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6620804Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6621106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6621179Z layer_outputs = layer_module( 2025-12-04T09:46:16.6621499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6621630Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6621940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6622082Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6622395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6622515Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6622518Z 2025-12-04T09:46:16.6622637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6622846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6622914Z res = mod(**inputs) 2025-12-04T09:46:16.6623233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6623311Z outputs = self.mobilebert( 2025-12-04T09:46:16.6623625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6623703Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6624026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6624114Z layer_outputs = layer_module( 2025-12-04T09:46:16.6624432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6624542Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6624863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6625004Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6625331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6625421Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6625425Z 2025-12-04T09:46:16.6625534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6625782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6625857Z res = mod(**inputs) 2025-12-04T09:46:16.6626184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6626260Z outputs = self.mobilebert( 2025-12-04T09:46:16.6626570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6626659Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6626988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6627072Z layer_outputs = layer_module( 2025-12-04T09:46:16.6627405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6627509Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6627834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6627970Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6628276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6628419Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6628745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6628853Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6628857Z 2025-12-04T09:46:16.6628969Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6629185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6629265Z res = mod(**inputs) 2025-12-04T09:46:16.6629580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6629661Z outputs = self.mobilebert( 2025-12-04T09:46:16.6629957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6630032Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6630346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6630425Z layer_outputs = layer_module( 2025-12-04T09:46:16.6630732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6630841Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6631147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6631279Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6631583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6631675Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6631679Z 2025-12-04T09:46:16.6631798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6632016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6632092Z res = mod(**inputs) 2025-12-04T09:46:16.6632400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6632477Z outputs = self.mobilebert( 2025-12-04T09:46:16.6632817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6632899Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6633220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6633295Z layer_outputs = layer_module( 2025-12-04T09:46:16.6633604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6633737Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6634045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6634180Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6634503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6634626Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6634630Z 2025-12-04T09:46:16.6634749Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6634963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6635034Z res = mod(**inputs) 2025-12-04T09:46:16.6635346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6635448Z outputs = self.mobilebert( 2025-12-04T09:46:16.6635769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6635849Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6636161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6636244Z layer_outputs = layer_module( 2025-12-04T09:46:16.6636558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6636658Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6636979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6637117Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6637433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6637527Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6637530Z 2025-12-04T09:46:16.6637642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6637871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6637941Z res = mod(**inputs) 2025-12-04T09:46:16.6638257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6638334Z outputs = self.mobilebert( 2025-12-04T09:46:16.6638646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6638734Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6639049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6639124Z layer_outputs = layer_module( 2025-12-04T09:46:16.6639444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6639565Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6639877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6640013Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6640319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6640538Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6640873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6640981Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6640985Z 2025-12-04T09:46:16.6641123Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6641344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6641426Z res = mod(**inputs) 2025-12-04T09:46:16.6641730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6641816Z outputs = self.mobilebert( 2025-12-04T09:46:16.6642121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6642223Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6642537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6642615Z layer_outputs = layer_module( 2025-12-04T09:46:16.6642923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6643036Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6643345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6643481Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6643788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6643880Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6643885Z 2025-12-04T09:46:16.6644008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6644225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6644304Z res = mod(**inputs) 2025-12-04T09:46:16.6644611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6644691Z outputs = self.mobilebert( 2025-12-04T09:46:16.6645007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6645086Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6645389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6645475Z layer_outputs = layer_module( 2025-12-04T09:46:16.6645782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6645892Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6646198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6646321Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6646656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6646782Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6646786Z 2025-12-04T09:46:16.6646910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6647320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6647406Z res = mod(**inputs) 2025-12-04T09:46:16.6647723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6647849Z outputs = self.mobilebert( 2025-12-04T09:46:16.6648161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6648269Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6648578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6648661Z layer_outputs = layer_module( 2025-12-04T09:46:16.6648959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6649057Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6649361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6649534Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6649843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6649935Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6649939Z 2025-12-04T09:46:16.6650049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6650270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6650339Z res = mod(**inputs) 2025-12-04T09:46:16.6650645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6650719Z outputs = self.mobilebert( 2025-12-04T09:46:16.6651020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6651108Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6651406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6651481Z layer_outputs = layer_module( 2025-12-04T09:46:16.6651795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6651900Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6652215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6652350Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6652658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6652804Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6653113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6653218Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6653222Z 2025-12-04T09:46:16.6653335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6653584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6653663Z res = mod(**inputs) 2025-12-04T09:46:16.6653971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6654049Z outputs = self.mobilebert( 2025-12-04T09:46:16.6654362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6654442Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6654771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6654848Z layer_outputs = layer_module( 2025-12-04T09:46:16.6655172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6655318Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6655632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6655727Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6655730Z 2025-12-04T09:46:16.6655839Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6656047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6656148Z res = mod(**inputs) 2025-12-04T09:46:16.6656453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6656535Z outputs = self.mobilebert( 2025-12-04T09:46:16.6656844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6656924Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6657248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6657325Z layer_outputs = layer_module( 2025-12-04T09:46:16.6657644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6657782Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6658090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6658219Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6658223Z 2025-12-04T09:46:16.6658331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6658543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6658620Z res = mod(**inputs) 2025-12-04T09:46:16.6658934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6659016Z outputs = self.mobilebert( 2025-12-04T09:46:16.6659332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6659409Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6659724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6659801Z layer_outputs = layer_module( 2025-12-04T09:46:16.6660106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6660286Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6660619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6660735Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6660743Z 2025-12-04T09:46:16.6660855Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6661071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6661149Z res = mod(**inputs) 2025-12-04T09:46:16.6661463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6661572Z outputs = self.mobilebert( 2025-12-04T09:46:16.6661911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6661992Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6662304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6662382Z layer_outputs = layer_module( 2025-12-04T09:46:16.6662694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6662868Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6663180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6663341Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6663637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6663736Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6663740Z 2025-12-04T09:46:16.6663858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6664068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6664145Z res = mod(**inputs) 2025-12-04T09:46:16.6664453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6664529Z outputs = self.mobilebert( 2025-12-04T09:46:16.6664836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6664915Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6665221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6665298Z layer_outputs = layer_module( 2025-12-04T09:46:16.6665597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6665769Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6666076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6666206Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6666519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6666612Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6666616Z 2025-12-04T09:46:16.6666729Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6666940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6667008Z res = mod(**inputs) 2025-12-04T09:46:16.6667336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6667413Z outputs = self.mobilebert( 2025-12-04T09:46:16.6667738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6667817Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6668126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6668230Z layer_outputs = layer_module( 2025-12-04T09:46:16.6668537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6668724Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6669045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6669174Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6669475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6669603Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6669901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6670030Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6670034Z 2025-12-04T09:46:16.6670146Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6670371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6670442Z res = mod(**inputs) 2025-12-04T09:46:16.6670750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6670835Z outputs = self.mobilebert( 2025-12-04T09:46:16.6671141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6671227Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6671532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6671611Z layer_outputs = layer_module( 2025-12-04T09:46:16.6671921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6672100Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6672407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6672536Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6672840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6672934Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6672938Z 2025-12-04T09:46:16.6673047Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6673266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6673346Z res = mod(**inputs) 2025-12-04T09:46:16.6673649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6673732Z outputs = self.mobilebert( 2025-12-04T09:46:16.6674059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6674140Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6674459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6674535Z layer_outputs = layer_module( 2025-12-04T09:46:16.6674848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6674951Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6675278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6675362Z self_outputs = self.self( 2025-12-04T09:46:16.6675692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6675774Z self.value(value_tensor) 2025-12-04T09:46:16.6675779Z 2025-12-04T09:46:16.6675901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6676122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6676199Z res = mod(**inputs) 2025-12-04T09:46:16.6676513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6676612Z outputs = self.mobilebert( 2025-12-04T09:46:16.6676930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6677011Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6677320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6677404Z layer_outputs = layer_module( 2025-12-04T09:46:16.6677713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6677897Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6678207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6678331Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6678648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6678740Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6678744Z 2025-12-04T09:46:16.6678864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6679083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6679156Z res = mod(**inputs) 2025-12-04T09:46:16.6679470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6679547Z outputs = self.mobilebert( 2025-12-04T09:46:16.6679870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6679949Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6680258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6680410Z layer_outputs = layer_module( 2025-12-04T09:46:16.6680746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6680924Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6681273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6681398Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6681714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6681812Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6682121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6682254Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6682258Z 2025-12-04T09:46:16.6682372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6682614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6682688Z res = mod(**inputs) 2025-12-04T09:46:16.6683067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6683158Z outputs = self.mobilebert( 2025-12-04T09:46:16.6683475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6683556Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6683882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6683984Z layer_outputs = layer_module( 2025-12-04T09:46:16.6684295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6684391Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6684695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6684782Z self_outputs = self.self( 2025-12-04T09:46:16.6685097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6685184Z self.query(query_tensor) 2025-12-04T09:46:16.6685188Z 2025-12-04T09:46:16.6685299Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6685521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6685598Z res = mod(**inputs) 2025-12-04T09:46:16.6685892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6685966Z outputs = self.mobilebert( 2025-12-04T09:46:16.6686281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6686359Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6686670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6686744Z layer_outputs = layer_module( 2025-12-04T09:46:16.6687050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6687151Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6687450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6687533Z self_outputs = self.self( 2025-12-04T09:46:16.6687840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6687913Z self.key(key_tensor) 2025-12-04T09:46:16.6687917Z 2025-12-04T09:46:16.6688030Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6688116Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6688225Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6688442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6688510Z res = mod(**inputs) 2025-12-04T09:46:16.6688825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6688920Z outputs = self.mobilebert( 2025-12-04T09:46:16.6689222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6689308Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6689626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6689706Z layer_outputs = layer_module( 2025-12-04T09:46:16.6690011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6690102Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6690408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6690542Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6690875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6690976Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6690980Z 2025-12-04T09:46:16.6691095Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6691318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6691389Z res = mod(**inputs) 2025-12-04T09:46:16.6691694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6691779Z outputs = self.mobilebert( 2025-12-04T09:46:16.6692083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6692170Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6692483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6692559Z layer_outputs = layer_module( 2025-12-04T09:46:16.6692862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6692953Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6693252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6693387Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6693683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6693824Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6694123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6694226Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6694230Z 2025-12-04T09:46:16.6694352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6694571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6694649Z res = mod(**inputs) 2025-12-04T09:46:16.6694975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6695056Z outputs = self.mobilebert( 2025-12-04T09:46:16.6695369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6695448Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6695767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6695873Z layer_outputs = layer_module( 2025-12-04T09:46:16.6696190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6696320Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6696625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6696751Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6697069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6697161Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6697165Z 2025-12-04T09:46:16.6697285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6697522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6697593Z res = mod(**inputs) 2025-12-04T09:46:16.6697907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6697985Z outputs = self.mobilebert( 2025-12-04T09:46:16.6698293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6698380Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6698693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6698776Z layer_outputs = layer_module( 2025-12-04T09:46:16.6699073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6699177Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6699482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6699602Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6699908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6700029Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6700033Z 2025-12-04T09:46:16.6700141Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6700359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6700427Z res = mod(**inputs) 2025-12-04T09:46:16.6700733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6700812Z outputs = self.mobilebert( 2025-12-04T09:46:16.6701110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6701195Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6701496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6701591Z layer_outputs = layer_module( 2025-12-04T09:46:16.6701896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6701997Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6702303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6702440Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6702764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6702864Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6702868Z 2025-12-04T09:46:16.6702999Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6703225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6703295Z res = mod(**inputs) 2025-12-04T09:46:16.6703600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6703686Z outputs = self.mobilebert( 2025-12-04T09:46:16.6703992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6704090Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6704417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6704496Z layer_outputs = layer_module( 2025-12-04T09:46:16.6704819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6704922Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6705241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6705389Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6705704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6705846Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6706165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6706266Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6706270Z 2025-12-04T09:46:16.6706389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6706611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6706681Z res = mod(**inputs) 2025-12-04T09:46:16.6707004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6707081Z outputs = self.mobilebert( 2025-12-04T09:46:16.6707403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6707482Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6707799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6707886Z layer_outputs = layer_module( 2025-12-04T09:46:16.6708198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6708307Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6708645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6708769Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6709082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6709175Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6709179Z 2025-12-04T09:46:16.6709299Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6709538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6709607Z res = mod(**inputs) 2025-12-04T09:46:16.6709935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6710013Z outputs = self.mobilebert( 2025-12-04T09:46:16.6710321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6710409Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6710717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6710799Z layer_outputs = layer_module( 2025-12-04T09:46:16.6711107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6711230Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6711545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6711669Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6711986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6712109Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6712113Z 2025-12-04T09:46:16.6712225Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6712447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6712517Z res = mod(**inputs) 2025-12-04T09:46:16.6712825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6712912Z outputs = self.mobilebert( 2025-12-04T09:46:16.6713218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6713304Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6713614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6713690Z layer_outputs = layer_module( 2025-12-04T09:46:16.6714008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6714108Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6714423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6714563Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6714873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6714975Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6714979Z 2025-12-04T09:46:16.6715092Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6715357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6715430Z res = mod(**inputs) 2025-12-04T09:46:16.6715736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6715822Z outputs = self.mobilebert( 2025-12-04T09:46:16.6716133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6716217Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6716555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6716643Z layer_outputs = layer_module( 2025-12-04T09:46:16.6716966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6717069Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6717363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6717503Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6717797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6717953Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6718251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6718347Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6718351Z 2025-12-04T09:46:16.6718480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6718688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6718756Z res = mod(**inputs) 2025-12-04T09:46:16.6719059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6719134Z outputs = self.mobilebert( 2025-12-04T09:46:16.6719437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6719515Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6719811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6719892Z layer_outputs = layer_module( 2025-12-04T09:46:16.6720195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6720303Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6720700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6720830Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6721156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6721249Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6721255Z 2025-12-04T09:46:16.6721369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6721596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6721667Z res = mod(**inputs) 2025-12-04T09:46:16.6721996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6722074Z outputs = self.mobilebert( 2025-12-04T09:46:16.6722448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6722540Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6722861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6722950Z layer_outputs = layer_module( 2025-12-04T09:46:16.6723273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6723395Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6723710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6723851Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6724162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6724292Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6724297Z 2025-12-04T09:46:16.6724411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6724635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6724706Z res = mod(**inputs) 2025-12-04T09:46:16.6725037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6725135Z outputs = self.mobilebert( 2025-12-04T09:46:16.6725429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6725515Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6725815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6725889Z layer_outputs = layer_module( 2025-12-04T09:46:16.6726193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6726291Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6726598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6726745Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6727038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6727134Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6727139Z 2025-12-04T09:46:16.6727246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6727455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6727529Z res = mod(**inputs) 2025-12-04T09:46:16.6727833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6727914Z outputs = self.mobilebert( 2025-12-04T09:46:16.6728219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6728296Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6728599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6728672Z layer_outputs = layer_module( 2025-12-04T09:46:16.6728984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6729109Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6729409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6729549Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6729853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6729987Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6730314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6730412Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6730416Z 2025-12-04T09:46:16.6730547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6730759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6730827Z res = mod(**inputs) 2025-12-04T09:46:16.6731142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6731218Z outputs = self.mobilebert( 2025-12-04T09:46:16.6731524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6731631Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6731936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6732026Z layer_outputs = layer_module( 2025-12-04T09:46:16.6732333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6732471Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6732790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6732887Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6732891Z 2025-12-04T09:46:16.6733016Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6733234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6733311Z res = mod(**inputs) 2025-12-04T09:46:16.6733629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6733710Z outputs = self.mobilebert( 2025-12-04T09:46:16.6734026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6734109Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6734416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6734505Z layer_outputs = layer_module( 2025-12-04T09:46:16.6734813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6734947Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6735266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6735394Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6735397Z 2025-12-04T09:46:16.6735519Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6735741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6735837Z res = mod(**inputs) 2025-12-04T09:46:16.6736157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6736235Z outputs = self.mobilebert( 2025-12-04T09:46:16.6736546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6736624Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6736931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6737036Z layer_outputs = layer_module( 2025-12-04T09:46:16.6737344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6737540Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6737854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6737960Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6737964Z 2025-12-04T09:46:16.6738084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6738301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6738371Z res = mod(**inputs) 2025-12-04T09:46:16.6738703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6738782Z outputs = self.mobilebert( 2025-12-04T09:46:16.6739103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6739182Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6739490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6739575Z layer_outputs = layer_module( 2025-12-04T09:46:16.6739883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6740064Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6740371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6740510Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6740824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6740926Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6740930Z 2025-12-04T09:46:16.6741052Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6741268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6741338Z res = mod(**inputs) 2025-12-04T09:46:16.6741647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6741723Z outputs = self.mobilebert( 2025-12-04T09:46:16.6742052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6742143Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6742451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6742538Z layer_outputs = layer_module( 2025-12-04T09:46:16.6742870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6743043Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6743358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6743494Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6743817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6743934Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6743938Z 2025-12-04T09:46:16.6744050Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6744291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6744364Z res = mod(**inputs) 2025-12-04T09:46:16.6744677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6744762Z outputs = self.mobilebert( 2025-12-04T09:46:16.6745066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6745151Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6745472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6745569Z layer_outputs = layer_module( 2025-12-04T09:46:16.6745884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6746056Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6746382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6746519Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6746825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6746967Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6747463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6747584Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6747588Z 2025-12-04T09:46:16.6747700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6747919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6748000Z res = mod(**inputs) 2025-12-04T09:46:16.6748304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6748384Z outputs = self.mobilebert( 2025-12-04T09:46:16.6748701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6748780Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6749103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6749183Z layer_outputs = layer_module( 2025-12-04T09:46:16.6749492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6749682Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6750061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6750194Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6750517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6750603Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6750607Z 2025-12-04T09:46:16.6750725Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6750942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6751045Z res = mod(**inputs) 2025-12-04T09:46:16.6751365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6751469Z outputs = self.mobilebert( 2025-12-04T09:46:16.6751792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6751874Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6752185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6752272Z layer_outputs = layer_module( 2025-12-04T09:46:16.6752580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6752720Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6753048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6753125Z self_outputs = self.self( 2025-12-04T09:46:16.6753430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6753510Z self.value(value_tensor) 2025-12-04T09:46:16.6753514Z 2025-12-04T09:46:16.6753626Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6753844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6753913Z res = mod(**inputs) 2025-12-04T09:46:16.6754217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6754295Z outputs = self.mobilebert( 2025-12-04T09:46:16.6754591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6754680Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6754977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6755061Z layer_outputs = layer_module( 2025-12-04T09:46:16.6755359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6755529Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6755833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6755953Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6756259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6756349Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6756353Z 2025-12-04T09:46:16.6756463Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6756678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6756746Z res = mod(**inputs) 2025-12-04T09:46:16.6757064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6757149Z outputs = self.mobilebert( 2025-12-04T09:46:16.6757447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6757531Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6757831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6757927Z layer_outputs = layer_module( 2025-12-04T09:46:16.6758236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6758425Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6758734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6758851Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6759147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6759249Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6759543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6759665Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6759677Z 2025-12-04T09:46:16.6759788Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6760000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6760077Z res = mod(**inputs) 2025-12-04T09:46:16.6760446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6760533Z outputs = self.mobilebert( 2025-12-04T09:46:16.6760847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6760926Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6761239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6761321Z layer_outputs = layer_module( 2025-12-04T09:46:16.6761632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6761732Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6762030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6762107Z self_outputs = self.self( 2025-12-04T09:46:16.6762412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6762488Z self.query(query_tensor) 2025-12-04T09:46:16.6762491Z 2025-12-04T09:46:16.6762608Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6762815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6762889Z res = mod(**inputs) 2025-12-04T09:46:16.6763196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6763270Z outputs = self.mobilebert( 2025-12-04T09:46:16.6763588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6763695Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6763994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6764078Z layer_outputs = layer_module( 2025-12-04T09:46:16.6764389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6764480Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6764808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6764883Z self_outputs = self.self( 2025-12-04T09:46:16.6765214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6765290Z self.key(key_tensor) 2025-12-04T09:46:16.6765293Z 2025-12-04T09:46:16.6765384Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6765478Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6765586Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6765794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6765870Z res = mod(**inputs) 2025-12-04T09:46:16.6766172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6766276Z outputs = self.mobilebert( 2025-12-04T09:46:16.6766585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6766661Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6766985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6767061Z layer_outputs = layer_module( 2025-12-04T09:46:16.6767379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6767471Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6767786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6767926Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6768239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6768329Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6768341Z 2025-12-04T09:46:16.6768453Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6768669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6768746Z res = mod(**inputs) 2025-12-04T09:46:16.6769055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6769131Z outputs = self.mobilebert( 2025-12-04T09:46:16.6769457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6769534Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6769850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6769926Z layer_outputs = layer_module( 2025-12-04T09:46:16.6770236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6770336Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6770664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6770806Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6771115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6771249Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6771556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6771678Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6771682Z 2025-12-04T09:46:16.6771792Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6772027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6772098Z res = mod(**inputs) 2025-12-04T09:46:16.6772410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6772485Z outputs = self.mobilebert( 2025-12-04T09:46:16.6772798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6772882Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6773182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6773286Z layer_outputs = layer_module( 2025-12-04T09:46:16.6773589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6773695Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6774013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6774138Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6774450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6774549Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6774553Z 2025-12-04T09:46:16.6774665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6774899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6774970Z res = mod(**inputs) 2025-12-04T09:46:16.6775269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6775356Z outputs = self.mobilebert( 2025-12-04T09:46:16.6775667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6775754Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6776066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6776143Z layer_outputs = layer_module( 2025-12-04T09:46:16.6776460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6776567Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6776878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6777008Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6777320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6777471Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6777476Z 2025-12-04T09:46:16.6777591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6777807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6777886Z res = mod(**inputs) 2025-12-04T09:46:16.6778193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6778298Z outputs = self.mobilebert( 2025-12-04T09:46:16.6778602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6778680Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6779025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6779105Z layer_outputs = layer_module( 2025-12-04T09:46:16.6779416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6779526Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6779834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6780000Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6780307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6780400Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6780404Z 2025-12-04T09:46:16.6780524Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6780738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6780815Z res = mod(**inputs) 2025-12-04T09:46:16.6781119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6781195Z outputs = self.mobilebert( 2025-12-04T09:46:16.6781506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6781585Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6781898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6781974Z layer_outputs = layer_module( 2025-12-04T09:46:16.6782281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6782390Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6782697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6782835Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6783151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6783282Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6783598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6783698Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6783702Z 2025-12-04T09:46:16.6783816Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6784042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6784139Z res = mod(**inputs) 2025-12-04T09:46:16.6784456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6784534Z outputs = self.mobilebert( 2025-12-04T09:46:16.6784864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6784952Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6785259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6785360Z layer_outputs = layer_module( 2025-12-04T09:46:16.6785687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6785811Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6786147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6786271Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6786582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6786683Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6786686Z 2025-12-04T09:46:16.6786818Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6787044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6787115Z res = mod(**inputs) 2025-12-04T09:46:16.6787445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6787532Z outputs = self.mobilebert( 2025-12-04T09:46:16.6787864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6787952Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6788281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6788358Z layer_outputs = layer_module( 2025-12-04T09:46:16.6788690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6788796Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6789101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6789235Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6789555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6789689Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6789692Z 2025-12-04T09:46:16.6789805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6790024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6790114Z res = mod(**inputs) 2025-12-04T09:46:16.6790437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6790525Z outputs = self.mobilebert( 2025-12-04T09:46:16.6790821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6790898Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6791231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6791308Z layer_outputs = layer_module( 2025-12-04T09:46:16.6791618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6791730Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6792048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6792194Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6792523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6792617Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6792621Z 2025-12-04T09:46:16.6792762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6792984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6793061Z res = mod(**inputs) 2025-12-04T09:46:16.6793375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6793453Z outputs = self.mobilebert( 2025-12-04T09:46:16.6793769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6793869Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6794179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6794263Z layer_outputs = layer_module( 2025-12-04T09:46:16.6794573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6794685Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6794993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6795131Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6795445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6795580Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6795901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6796001Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6796004Z 2025-12-04T09:46:16.6796117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6796347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6796418Z res = mod(**inputs) 2025-12-04T09:46:16.6796732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6796810Z outputs = self.mobilebert( 2025-12-04T09:46:16.6797117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6797202Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6797510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6797585Z layer_outputs = layer_module( 2025-12-04T09:46:16.6797900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6798001Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6798333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6798457Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6798765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6798863Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6798868Z 2025-12-04T09:46:16.6798979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6799223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6799294Z res = mod(**inputs) 2025-12-04T09:46:16.6799621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6799708Z outputs = self.mobilebert( 2025-12-04T09:46:16.6800017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6800095Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6800490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6800573Z layer_outputs = layer_module( 2025-12-04T09:46:16.6800895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6801021Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6801327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6801459Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6801768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6801901Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6801915Z 2025-12-04T09:46:16.6802024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6802235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6802311Z res = mod(**inputs) 2025-12-04T09:46:16.6802610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6802690Z outputs = self.mobilebert( 2025-12-04T09:46:16.6802997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6803075Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6803383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6803459Z layer_outputs = layer_module( 2025-12-04T09:46:16.6803759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6803867Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6804177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6804322Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6804621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6804712Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6804717Z 2025-12-04T09:46:16.6804834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6805063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6805141Z res = mod(**inputs) 2025-12-04T09:46:16.6805437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6805515Z outputs = self.mobilebert( 2025-12-04T09:46:16.6805816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6805922Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6806224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6806309Z layer_outputs = layer_module( 2025-12-04T09:46:16.6806638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6806750Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6807072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6807208Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6807514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6807661Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6807969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6808065Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6808068Z 2025-12-04T09:46:16.6808179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6808400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6808467Z res = mod(**inputs) 2025-12-04T09:46:16.6808762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6808846Z outputs = self.mobilebert( 2025-12-04T09:46:16.6809154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6809238Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6809535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6809609Z layer_outputs = layer_module( 2025-12-04T09:46:16.6809913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6810045Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6810358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6810447Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6810450Z 2025-12-04T09:46:16.6810557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6810773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6810842Z res = mod(**inputs) 2025-12-04T09:46:16.6811141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6811224Z outputs = self.mobilebert( 2025-12-04T09:46:16.6811535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6811618Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6811948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6812024Z layer_outputs = layer_module( 2025-12-04T09:46:16.6812345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6812474Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6812784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6812928Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6812931Z 2025-12-04T09:46:16.6813041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6813276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6813347Z res = mod(**inputs) 2025-12-04T09:46:16.6813656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6813730Z outputs = self.mobilebert( 2025-12-04T09:46:16.6814030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6814113Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6814433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6814510Z layer_outputs = layer_module( 2025-12-04T09:46:16.6814824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6815001Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6815320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6815423Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6815427Z 2025-12-04T09:46:16.6815539Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6815773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6815839Z res = mod(**inputs) 2025-12-04T09:46:16.6816141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6816217Z outputs = self.mobilebert( 2025-12-04T09:46:16.6816512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6816597Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6816916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6816992Z layer_outputs = layer_module( 2025-12-04T09:46:16.6817306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6817479Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6817788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6817925Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6818233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6818344Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6818348Z 2025-12-04T09:46:16.6818477Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6818709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6818775Z res = mod(**inputs) 2025-12-04T09:46:16.6819072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6819155Z outputs = self.mobilebert( 2025-12-04T09:46:16.6819452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6819549Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6819864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6819958Z layer_outputs = layer_module( 2025-12-04T09:46:16.6820279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6820449Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6820758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6820904Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6821209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6821334Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6821338Z 2025-12-04T09:46:16.6821452Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6821672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6821750Z res = mod(**inputs) 2025-12-04T09:46:16.6822063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6822150Z outputs = self.mobilebert( 2025-12-04T09:46:16.6822457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6822536Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6822849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6822929Z layer_outputs = layer_module( 2025-12-04T09:46:16.6823237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6823418Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6823725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6823867Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6824174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6824305Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6824623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6824725Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6824729Z 2025-12-04T09:46:16.6824848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6825064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6825136Z res = mod(**inputs) 2025-12-04T09:46:16.6825479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6825559Z outputs = self.mobilebert( 2025-12-04T09:46:16.6825884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6825964Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6826282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6826392Z layer_outputs = layer_module( 2025-12-04T09:46:16.6826705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6826898Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6827217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6827338Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6827662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6827753Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6827757Z 2025-12-04T09:46:16.6827870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6828147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6828221Z res = mod(**inputs) 2025-12-04T09:46:16.6828536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6828614Z outputs = self.mobilebert( 2025-12-04T09:46:16.6828923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6829010Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6829317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6829393Z layer_outputs = layer_module( 2025-12-04T09:46:16.6829716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6829812Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6830126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6830203Z self_outputs = self.self( 2025-12-04T09:46:16.6830522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6830609Z self.value(value_tensor) 2025-12-04T09:46:16.6830614Z 2025-12-04T09:46:16.6830726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6830948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6831018Z res = mod(**inputs) 2025-12-04T09:46:16.6831338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6831424Z outputs = self.mobilebert( 2025-12-04T09:46:16.6831738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6831821Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6832140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6832218Z layer_outputs = layer_module( 2025-12-04T09:46:16.6832576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6832754Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6833071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6833201Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6833510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6833628Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6833632Z 2025-12-04T09:46:16.6833745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6833978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6834059Z res = mod(**inputs) 2025-12-04T09:46:16.6834368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6834468Z outputs = self.mobilebert( 2025-12-04T09:46:16.6834794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6834872Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6835189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6835288Z layer_outputs = layer_module( 2025-12-04T09:46:16.6835595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6835776Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6836087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6836215Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6836524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6836618Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6836938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6837041Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6837044Z 2025-12-04T09:46:16.6837165Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6837382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6837454Z res = mod(**inputs) 2025-12-04T09:46:16.6837769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6837847Z outputs = self.mobilebert( 2025-12-04T09:46:16.6838159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6838245Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6838552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6838641Z layer_outputs = layer_module( 2025-12-04T09:46:16.6838949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6839047Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6839382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6839462Z self_outputs = self.self( 2025-12-04T09:46:16.6839776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6839855Z self.query(query_tensor) 2025-12-04T09:46:16.6839859Z 2025-12-04T09:46:16.6839974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6840197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6840288Z res = mod(**inputs) 2025-12-04T09:46:16.6840669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6840759Z outputs = self.mobilebert( 2025-12-04T09:46:16.6841094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6841189Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6841503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6841581Z layer_outputs = layer_module( 2025-12-04T09:46:16.6841902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6841998Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6842332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6842414Z self_outputs = self.self( 2025-12-04T09:46:16.6842725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6842810Z self.key(key_tensor) 2025-12-04T09:46:16.6842814Z 2025-12-04T09:46:16.6842905Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6842993Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6843118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6843334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6843415Z res = mod(**inputs) 2025-12-04T09:46:16.6843720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6843800Z outputs = self.mobilebert( 2025-12-04T09:46:16.6844116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6844194Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6844500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6844588Z layer_outputs = layer_module( 2025-12-04T09:46:16.6844894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6844995Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6845302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6845437Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6845756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6845848Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6845852Z 2025-12-04T09:46:16.6845971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6846186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6846278Z res = mod(**inputs) 2025-12-04T09:46:16.6846594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6846669Z outputs = self.mobilebert( 2025-12-04T09:46:16.6846991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6847337Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6847660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6847806Z layer_outputs = layer_module( 2025-12-04T09:46:16.6848142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6848239Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6848567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6848701Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6849013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6849151Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6849496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6849611Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6849615Z 2025-12-04T09:46:16.6849731Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6849959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6850030Z res = mod(**inputs) 2025-12-04T09:46:16.6850343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6850430Z outputs = self.mobilebert( 2025-12-04T09:46:16.6850736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6850812Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6851126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6851201Z layer_outputs = layer_module( 2025-12-04T09:46:16.6851505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6851611Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6851908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6852033Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6852339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6852435Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6852439Z 2025-12-04T09:46:16.6852550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6852762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6852840Z res = mod(**inputs) 2025-12-04T09:46:16.6853132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6853208Z outputs = self.mobilebert( 2025-12-04T09:46:16.6853552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6853631Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6853935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6854011Z layer_outputs = layer_module( 2025-12-04T09:46:16.6854318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6854452Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6854756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6854887Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6855222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6855351Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6855355Z 2025-12-04T09:46:16.6855475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6855691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6855761Z res = mod(**inputs) 2025-12-04T09:46:16.6856075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6856183Z outputs = self.mobilebert( 2025-12-04T09:46:16.6856505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6856582Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6856882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6856966Z layer_outputs = layer_module( 2025-12-04T09:46:16.6857264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6857371Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6857674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6857814Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6858131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6858235Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6858238Z 2025-12-04T09:46:16.6858358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6858570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6858641Z res = mod(**inputs) 2025-12-04T09:46:16.6858948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6859022Z outputs = self.mobilebert( 2025-12-04T09:46:16.6859324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6859411Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6859712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6859795Z layer_outputs = layer_module( 2025-12-04T09:46:16.6860094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6860193Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6860524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6860660Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6860968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6861099Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6861405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6861531Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6861534Z 2025-12-04T09:46:16.6861644Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6861873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6861968Z res = mod(**inputs) 2025-12-04T09:46:16.6862264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6862347Z outputs = self.mobilebert( 2025-12-04T09:46:16.6862642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6862719Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6863046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6863122Z layer_outputs = layer_module( 2025-12-04T09:46:16.6863434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6863539Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6863847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6863979Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6864286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6864378Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6864389Z 2025-12-04T09:46:16.6864513Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6864724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6864797Z res = mod(**inputs) 2025-12-04T09:46:16.6865092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6865170Z outputs = self.mobilebert( 2025-12-04T09:46:16.6865477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6865555Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6865868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6865944Z layer_outputs = layer_module( 2025-12-04T09:46:16.6866252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6866365Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6866674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6866805Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6867133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6867260Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6867263Z 2025-12-04T09:46:16.6867384Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6867601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6867671Z res = mod(**inputs) 2025-12-04T09:46:16.6867986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6868087Z outputs = self.mobilebert( 2025-12-04T09:46:16.6868400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6868480Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6868831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6868919Z layer_outputs = layer_module( 2025-12-04T09:46:16.6869227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6869335Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6869642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6869800Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6870113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6870207Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6870210Z 2025-12-04T09:46:16.6870323Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6870549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6870618Z res = mod(**inputs) 2025-12-04T09:46:16.6870930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6871009Z outputs = self.mobilebert( 2025-12-04T09:46:16.6871314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6871401Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6871708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6871794Z layer_outputs = layer_module( 2025-12-04T09:46:16.6872097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6872200Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6872517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6872655Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6872960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6873103Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6873411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6873517Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6873521Z 2025-12-04T09:46:16.6873636Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6873852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6873955Z res = mod(**inputs) 2025-12-04T09:46:16.6874264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6874349Z outputs = self.mobilebert( 2025-12-04T09:46:16.6874655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6874733Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6875046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6875146Z layer_outputs = layer_module( 2025-12-04T09:46:16.6875477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6875580Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6875886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6876016Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6876323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6876416Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6876447Z 2025-12-04T09:46:16.6876563Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6876787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6876864Z res = mod(**inputs) 2025-12-04T09:46:16.6877174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6877253Z outputs = self.mobilebert( 2025-12-04T09:46:16.6877569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6877648Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6877961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6878037Z layer_outputs = layer_module( 2025-12-04T09:46:16.6878345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6878455Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6878755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6878875Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6879185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6879305Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6879309Z 2025-12-04T09:46:16.6879424Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6879635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6879702Z res = mod(**inputs) 2025-12-04T09:46:16.6880008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6880087Z outputs = self.mobilebert( 2025-12-04T09:46:16.6880617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6880707Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6881040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6881130Z layer_outputs = layer_module( 2025-12-04T09:46:16.6881445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6881548Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6881876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6882032Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6882340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6882433Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6882467Z 2025-12-04T09:46:16.6882580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6882805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6882873Z res = mod(**inputs) 2025-12-04T09:46:16.6883176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6883254Z outputs = self.mobilebert( 2025-12-04T09:46:16.6883552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6883662Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6883961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6884035Z layer_outputs = layer_module( 2025-12-04T09:46:16.6884342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6884444Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6884749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6884881Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6885179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6885321Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6885624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6885730Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6885734Z 2025-12-04T09:46:16.6885845Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6886057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6886134Z res = mod(**inputs) 2025-12-04T09:46:16.6886432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6886515Z outputs = self.mobilebert( 2025-12-04T09:46:16.6886813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6886890Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6887201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6887275Z layer_outputs = layer_module( 2025-12-04T09:46:16.6887586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6887743Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6888044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6888142Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6888145Z 2025-12-04T09:46:16.6888254Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6888465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6888543Z res = mod(**inputs) 2025-12-04T09:46:16.6888865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6888948Z outputs = self.mobilebert( 2025-12-04T09:46:16.6889342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6889422Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6889730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6889803Z layer_outputs = layer_module( 2025-12-04T09:46:16.6890111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6890249Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6890568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6890697Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6890701Z 2025-12-04T09:46:16.6890813Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6891034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6891111Z res = mod(**inputs) 2025-12-04T09:46:16.6891419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6891503Z outputs = self.mobilebert( 2025-12-04T09:46:16.6891810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6891899Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6892206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6892284Z layer_outputs = layer_module( 2025-12-04T09:46:16.6892582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6892759Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6893066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6893176Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6893179Z 2025-12-04T09:46:16.6893288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6893498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6893575Z res = mod(**inputs) 2025-12-04T09:46:16.6893878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6893963Z outputs = self.mobilebert( 2025-12-04T09:46:16.6894271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6894355Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6894692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6894772Z layer_outputs = layer_module( 2025-12-04T09:46:16.6895096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6895270Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6895588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6895755Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6896078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6896199Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6896210Z 2025-12-04T09:46:16.6896327Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6896545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6896623Z res = mod(**inputs) 2025-12-04T09:46:16.6896940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6897019Z outputs = self.mobilebert( 2025-12-04T09:46:16.6897331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6897433Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6897756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6897837Z layer_outputs = layer_module( 2025-12-04T09:46:16.6898154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6898336Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6898652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6898795Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6899114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6899209Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6899213Z 2025-12-04T09:46:16.6899333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6899554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6899625Z res = mod(**inputs) 2025-12-04T09:46:16.6899948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6900025Z outputs = self.mobilebert( 2025-12-04T09:46:16.6900345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6900423Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6900740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6900828Z layer_outputs = layer_module( 2025-12-04T09:46:16.6901142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6901322Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6901658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6901799Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6902117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6902247Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6902555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6902689Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6902694Z 2025-12-04T09:46:16.6902807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6903030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6903118Z res = mod(**inputs) 2025-12-04T09:46:16.6903426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6903513Z outputs = self.mobilebert( 2025-12-04T09:46:16.6903817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6903902Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6904207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6904304Z layer_outputs = layer_module( 2025-12-04T09:46:16.6904630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6904808Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6905131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6905254Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6905568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6905669Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6905673Z 2025-12-04T09:46:16.6905784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6906005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6906084Z res = mod(**inputs) 2025-12-04T09:46:16.6906400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6906481Z outputs = self.mobilebert( 2025-12-04T09:46:16.6906799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6906876Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6907202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6907279Z layer_outputs = layer_module( 2025-12-04T09:46:16.6907603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6907699Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6908016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6908102Z self_outputs = self.self( 2025-12-04T09:46:16.6908421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6908499Z self.value(value_tensor) 2025-12-04T09:46:16.6908530Z 2025-12-04T09:46:16.6908647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6908864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6908941Z res = mod(**inputs) 2025-12-04T09:46:16.6909248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6909325Z outputs = self.mobilebert( 2025-12-04T09:46:16.6909640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6909756Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6910090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6910167Z layer_outputs = layer_module( 2025-12-04T09:46:16.6910465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6910644Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6910944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.6911064Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.6911397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6911488Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6911492Z 2025-12-04T09:46:16.6911610Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6911824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6911891Z res = mod(**inputs) 2025-12-04T09:46:16.6912199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6912276Z outputs = self.mobilebert( 2025-12-04T09:46:16.6912593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6912672Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6912981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6913069Z layer_outputs = layer_module( 2025-12-04T09:46:16.6913379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6913556Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6913877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6913997Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6914339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.6914437Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.6914743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6914855Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6914859Z 2025-12-04T09:46:16.6914971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6915196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6915265Z res = mod(**inputs) 2025-12-04T09:46:16.6915596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6915682Z outputs = self.mobilebert( 2025-12-04T09:46:16.6915982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6916068Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6916367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6916464Z layer_outputs = layer_module( 2025-12-04T09:46:16.6916780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6916877Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6917206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6917296Z self_outputs = self.self( 2025-12-04T09:46:16.6917602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.6917689Z self.query(query_tensor) 2025-12-04T09:46:16.6917693Z 2025-12-04T09:46:16.6917810Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6918029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6918140Z res = mod(**inputs) 2025-12-04T09:46:16.6918459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6918544Z outputs = self.mobilebert( 2025-12-04T09:46:16.6918859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6918942Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6919263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6919342Z layer_outputs = layer_module( 2025-12-04T09:46:16.6919657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6919758Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6920075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6920162Z self_outputs = self.self( 2025-12-04T09:46:16.6920557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.6920637Z self.key(key_tensor) 2025-12-04T09:46:16.6920642Z 2025-12-04T09:46:16.6920747Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6920839Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.6920955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6921182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6921256Z res = mod(**inputs) 2025-12-04T09:46:16.6921574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6921656Z outputs = self.mobilebert( 2025-12-04T09:46:16.6921964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6922064Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6922363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6922473Z layer_outputs = layer_module( 2025-12-04T09:46:16.6922780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6922874Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6923187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6923322Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6923654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.6923757Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6923761Z 2025-12-04T09:46:16.6923875Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6924125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6924198Z res = mod(**inputs) 2025-12-04T09:46:16.6924495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6924578Z outputs = self.mobilebert( 2025-12-04T09:46:16.6924886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6924971Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6925295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6925381Z layer_outputs = layer_module( 2025-12-04T09:46:16.6925683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6925774Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6926078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.6926220Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.6926525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.6926672Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6926980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6927083Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6927087Z 2025-12-04T09:46:16.6927206Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6927421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6927499Z res = mod(**inputs) 2025-12-04T09:46:16.6927805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6927882Z outputs = self.mobilebert( 2025-12-04T09:46:16.6928196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6928277Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6928602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6928684Z layer_outputs = layer_module( 2025-12-04T09:46:16.6928987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6929102Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6929427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6929554Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6929877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6929970Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6929973Z 2025-12-04T09:46:16.6930093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6930312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6930405Z res = mod(**inputs) 2025-12-04T09:46:16.6930722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6930818Z outputs = self.mobilebert( 2025-12-04T09:46:16.6931147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6931227Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6931540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6931627Z layer_outputs = layer_module( 2025-12-04T09:46:16.6931947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6932073Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6932391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6932516Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6932839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6932966Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6932969Z 2025-12-04T09:46:16.6933081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6933307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6933376Z res = mod(**inputs) 2025-12-04T09:46:16.6933696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6933790Z outputs = self.mobilebert( 2025-12-04T09:46:16.6934096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6934183Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6934505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6934583Z layer_outputs = layer_module( 2025-12-04T09:46:16.6934909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6935012Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6935333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6935472Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6935781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6935883Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6935886Z 2025-12-04T09:46:16.6936000Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6936274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6936348Z res = mod(**inputs) 2025-12-04T09:46:16.6936669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6936756Z outputs = self.mobilebert( 2025-12-04T09:46:16.6937072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6937159Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6937485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6937564Z layer_outputs = layer_module( 2025-12-04T09:46:16.6937931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6938037Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6938344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6938490Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6938796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6938937Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6939271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6939375Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6939378Z 2025-12-04T09:46:16.6939501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6939721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6939799Z res = mod(**inputs) 2025-12-04T09:46:16.6940107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6940186Z outputs = self.mobilebert( 2025-12-04T09:46:16.6940505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6940584Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6940893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6940977Z layer_outputs = layer_module( 2025-12-04T09:46:16.6941288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6941400Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6941712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6941837Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6942153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6942244Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6942248Z 2025-12-04T09:46:16.6942368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6942590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6942659Z res = mod(**inputs) 2025-12-04T09:46:16.6942973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6954885Z outputs = self.mobilebert( 2025-12-04T09:46:16.6955478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6955575Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6955896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6955993Z layer_outputs = layer_module( 2025-12-04T09:46:16.6956309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6956475Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6956797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6956927Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6957288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6957420Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6957428Z 2025-12-04T09:46:16.6957560Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6957793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6957871Z res = mod(**inputs) 2025-12-04T09:46:16.6958187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6958320Z outputs = self.mobilebert( 2025-12-04T09:46:16.6958633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6958726Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6959041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6959128Z layer_outputs = layer_module( 2025-12-04T09:46:16.6959441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6959546Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6959869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6960017Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6960423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6960545Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6960550Z 2025-12-04T09:46:16.6960673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6960906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6960980Z res = mod(**inputs) 2025-12-04T09:46:16.6961286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6961378Z outputs = self.mobilebert( 2025-12-04T09:46:16.6961684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6961778Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6962097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6962174Z layer_outputs = layer_module( 2025-12-04T09:46:16.6962484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6962608Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6962914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6963058Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6963373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6963508Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6963840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6963942Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6963946Z 2025-12-04T09:46:16.6964076Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6964299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6964369Z res = mod(**inputs) 2025-12-04T09:46:16.6964673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6964750Z outputs = self.mobilebert( 2025-12-04T09:46:16.6965047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6965135Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6965453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6965538Z layer_outputs = layer_module( 2025-12-04T09:46:16.6965839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6965939Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6966250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6966375Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6966677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6966778Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6966784Z 2025-12-04T09:46:16.6966894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6967115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6967185Z res = mod(**inputs) 2025-12-04T09:46:16.6967483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6967568Z outputs = self.mobilebert( 2025-12-04T09:46:16.6967866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6967952Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6968254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6968330Z layer_outputs = layer_module( 2025-12-04T09:46:16.6968636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6968738Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6969033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.6969162Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.6969477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6969608Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6969612Z 2025-12-04T09:46:16.6969723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6969936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6970013Z res = mod(**inputs) 2025-12-04T09:46:16.6970314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6970417Z outputs = self.mobilebert( 2025-12-04T09:46:16.6970717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6970813Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6971122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6971197Z layer_outputs = layer_module( 2025-12-04T09:46:16.6971493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6971600Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6971901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6972078Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6972378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.6972468Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6972474Z 2025-12-04T09:46:16.6972589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6972799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6972873Z res = mod(**inputs) 2025-12-04T09:46:16.6973168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6973245Z outputs = self.mobilebert( 2025-12-04T09:46:16.6973546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6973626Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6973931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6974006Z layer_outputs = layer_module( 2025-12-04T09:46:16.6974304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.6974410Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.6974706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.6974836Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.6975142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.6975273Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6975577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6975672Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6975676Z 2025-12-04T09:46:16.6975788Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6976017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6976090Z res = mod(**inputs) 2025-12-04T09:46:16.6976393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6976470Z outputs = self.mobilebert( 2025-12-04T09:46:16.6976769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6976857Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6977173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6977249Z layer_outputs = layer_module( 2025-12-04T09:46:16.6977576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6977714Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6978025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.6978116Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.6978120Z 2025-12-04T09:46:16.6978229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6978449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6978538Z res = mod(**inputs) 2025-12-04T09:46:16.6978839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6978915Z outputs = self.mobilebert( 2025-12-04T09:46:16.6979214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6979298Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6979598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6979674Z layer_outputs = layer_module( 2025-12-04T09:46:16.6979980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.6980110Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.6980413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.6980536Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.6980540Z 2025-12-04T09:46:16.6980650Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6980871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6980940Z res = mod(**inputs) 2025-12-04T09:46:16.6981247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6981323Z outputs = self.mobilebert( 2025-12-04T09:46:16.6981620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6981705Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6982013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6982097Z layer_outputs = layer_module( 2025-12-04T09:46:16.6982393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6982568Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6982903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.6983007Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.6983011Z 2025-12-04T09:46:16.6983122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6983344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6983415Z res = mod(**inputs) 2025-12-04T09:46:16.6983733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6983844Z outputs = self.mobilebert( 2025-12-04T09:46:16.6984147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6984253Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6984556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6984637Z layer_outputs = layer_module( 2025-12-04T09:46:16.6984940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6985113Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6985421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.6985579Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.6985882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6985989Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6985993Z 2025-12-04T09:46:16.6986106Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6986326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6986395Z res = mod(**inputs) 2025-12-04T09:46:16.6986696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6986781Z outputs = self.mobilebert( 2025-12-04T09:46:16.6987085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6987172Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6987483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6987562Z layer_outputs = layer_module( 2025-12-04T09:46:16.6987882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6988054Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6988368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6988505Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6988818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.6988922Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.6988926Z 2025-12-04T09:46:16.6989037Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6989255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6989333Z res = mod(**inputs) 2025-12-04T09:46:16.6989666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6989753Z outputs = self.mobilebert( 2025-12-04T09:46:16.6990052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6990128Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6990436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6990534Z layer_outputs = layer_module( 2025-12-04T09:46:16.6990839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.6991029Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.6991342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.6991485Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.6991791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.6991931Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.6992237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.6992360Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.6992364Z 2025-12-04T09:46:16.6992486Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6992700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6992772Z res = mod(**inputs) 2025-12-04T09:46:16.6993098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6993174Z outputs = self.mobilebert( 2025-12-04T09:46:16.6993477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6993554Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6993851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6993937Z layer_outputs = layer_module( 2025-12-04T09:46:16.6994231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.6994408Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.6994707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.6994832Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.6995146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.6995237Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.6995241Z 2025-12-04T09:46:16.6995352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6995575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6995646Z res = mod(**inputs) 2025-12-04T09:46:16.6995957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6996032Z outputs = self.mobilebert( 2025-12-04T09:46:16.6996367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6996458Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6996773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6996854Z layer_outputs = layer_module( 2025-12-04T09:46:16.6997167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.6997264Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.6997609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.6997688Z self_outputs = self.self( 2025-12-04T09:46:16.6998012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.6998100Z self.value(value_tensor) 2025-12-04T09:46:16.6998105Z 2025-12-04T09:46:16.6998216Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.6998435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.6998504Z res = mod(**inputs) 2025-12-04T09:46:16.6998800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.6998882Z outputs = self.mobilebert( 2025-12-04T09:46:16.6999211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.6999300Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.6999612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.6999689Z layer_outputs = layer_module( 2025-12-04T09:46:16.7000009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.7000186Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.7000589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.7000728Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.7001043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.7001148Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.7001152Z 2025-12-04T09:46:16.7001266Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7001496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7001575Z res = mod(**inputs) 2025-12-04T09:46:16.7001875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7001960Z outputs = self.mobilebert( 2025-12-04T09:46:16.7002261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7002339Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7002651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7002732Z layer_outputs = layer_module( 2025-12-04T09:46:16.7003036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.7003219Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.7003543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.7003672Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.7003973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.7004069Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.7004387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7004511Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7004515Z 2025-12-04T09:46:16.7004638Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7004878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7004951Z res = mod(**inputs) 2025-12-04T09:46:16.7005270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7005347Z outputs = self.mobilebert( 2025-12-04T09:46:16.7005662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7005742Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7006053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7006159Z layer_outputs = layer_module( 2025-12-04T09:46:16.7006463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7006558Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7006877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.7006955Z self_outputs = self.self( 2025-12-04T09:46:16.7007277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.7007357Z self.query(query_tensor) 2025-12-04T09:46:16.7007361Z 2025-12-04T09:46:16.7007474Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7007698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7007771Z res = mod(**inputs) 2025-12-04T09:46:16.7008093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7008169Z outputs = self.mobilebert( 2025-12-04T09:46:16.7008479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7008567Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7008877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7008956Z layer_outputs = layer_module( 2025-12-04T09:46:16.7009276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7009372Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7009693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.7009771Z self_outputs = self.self( 2025-12-04T09:46:16.7010083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.7010169Z self.key(key_tensor) 2025-12-04T09:46:16.7010173Z 2025-12-04T09:46:16.7010283Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.7010383Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.7010497Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7010714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7010791Z res = mod(**inputs) 2025-12-04T09:46:16.7011096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7011173Z outputs = self.mobilebert( 2025-12-04T09:46:16.7011515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7011594Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7012550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7012633Z layer_outputs = layer_module( 2025-12-04T09:46:16.7012957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7013060Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7013379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.7013517Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.7013856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.7013954Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7013957Z 2025-12-04T09:46:16.7014077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7014293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7014363Z res = mod(**inputs) 2025-12-04T09:46:16.7014679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7014755Z outputs = self.mobilebert( 2025-12-04T09:46:16.7015068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7015149Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7015460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7015545Z layer_outputs = layer_module( 2025-12-04T09:46:16.7015855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7015949Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7016267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.7016401Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.7016716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.7016853Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7017162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7017272Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7017276Z 2025-12-04T09:46:16.7017389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7017614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7017684Z res = mod(**inputs) 2025-12-04T09:46:16.7018011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7018100Z outputs = self.mobilebert( 2025-12-04T09:46:16.7018408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7018494Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7018803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7018904Z layer_outputs = layer_module( 2025-12-04T09:46:16.7019218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7019341Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7019648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7019782Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7020091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7020191Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7020195Z 2025-12-04T09:46:16.7020307Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7020543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7020625Z res = mod(**inputs) 2025-12-04T09:46:16.7020932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7021018Z outputs = self.mobilebert( 2025-12-04T09:46:16.7021326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7021406Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7021722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7021800Z layer_outputs = layer_module( 2025-12-04T09:46:16.7022106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7022220Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7022551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7022682Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7023018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7023146Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7023149Z 2025-12-04T09:46:16.7023270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7023483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7023559Z res = mod(**inputs) 2025-12-04T09:46:16.7023889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7023971Z outputs = self.mobilebert( 2025-12-04T09:46:16.7024305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7024386Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7024717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7024823Z layer_outputs = layer_module( 2025-12-04T09:46:16.7025158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7025269Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7025605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7025743Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7026116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.7026209Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7026213Z 2025-12-04T09:46:16.7026349Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7026566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7026638Z res = mod(**inputs) 2025-12-04T09:46:16.7026953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7027028Z outputs = self.mobilebert( 2025-12-04T09:46:16.7027369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7027449Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7027806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7027894Z layer_outputs = layer_module( 2025-12-04T09:46:16.7028229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7028333Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7028678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7028815Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7029199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.7029335Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7029657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7029766Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7029770Z 2025-12-04T09:46:16.7029882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7030107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7030179Z res = mod(**inputs) 2025-12-04T09:46:16.7030497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7030581Z outputs = self.mobilebert( 2025-12-04T09:46:16.7030903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7030984Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7031317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7031395Z layer_outputs = layer_module( 2025-12-04T09:46:16.7031716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7031821Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7032160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7032297Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7032601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7032696Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7032700Z 2025-12-04T09:46:16.7032811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7033046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7033123Z res = mod(**inputs) 2025-12-04T09:46:16.7033438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7033538Z outputs = self.mobilebert( 2025-12-04T09:46:16.7033848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7033926Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7034237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7034312Z layer_outputs = layer_module( 2025-12-04T09:46:16.7034629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7034764Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7035084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7035208Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7035527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7035657Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7035661Z 2025-12-04T09:46:16.7035773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7035985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7036062Z res = mod(**inputs) 2025-12-04T09:46:16.7036376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7036463Z outputs = self.mobilebert( 2025-12-04T09:46:16.7036772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7036851Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7037170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7037249Z layer_outputs = layer_module( 2025-12-04T09:46:16.7037573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7037677Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7037993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7038139Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7038450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.7038543Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7038552Z 2025-12-04T09:46:16.7038665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7038903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7038979Z res = mod(**inputs) 2025-12-04T09:46:16.7039286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7039364Z outputs = self.mobilebert( 2025-12-04T09:46:16.7039675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7039756Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7040087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7040163Z layer_outputs = layer_module( 2025-12-04T09:46:16.7040590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7040709Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7041028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7041165Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7041482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.7041614Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7041956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7042059Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7042063Z 2025-12-04T09:46:16.7042181Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7042410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7042482Z res = mod(**inputs) 2025-12-04T09:46:16.7042801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7042880Z outputs = self.mobilebert( 2025-12-04T09:46:16.7043190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7043278Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7043593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7043682Z layer_outputs = layer_module( 2025-12-04T09:46:16.7043993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7044096Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7044415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7044538Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7044847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7044949Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7044952Z 2025-12-04T09:46:16.7045066Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7045290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7045360Z res = mod(**inputs) 2025-12-04T09:46:16.7045674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7045760Z outputs = self.mobilebert( 2025-12-04T09:46:16.7046093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7046180Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7046485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7046564Z layer_outputs = layer_module( 2025-12-04T09:46:16.7046881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7047006Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7047465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7047657Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7047976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7048112Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7048117Z 2025-12-04T09:46:16.7048232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7048450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7048528Z res = mod(**inputs) 2025-12-04T09:46:16.7048841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7048963Z outputs = self.mobilebert( 2025-12-04T09:46:16.7049271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7049355Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7049678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7049758Z layer_outputs = layer_module( 2025-12-04T09:46:16.7050070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7050183Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7050493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7050643Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7050959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.7051054Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7051057Z 2025-12-04T09:46:16.7051184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7051403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7051484Z res = mod(**inputs) 2025-12-04T09:46:16.7051796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7051873Z outputs = self.mobilebert( 2025-12-04T09:46:16.7052187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7052269Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7052579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7052665Z layer_outputs = layer_module( 2025-12-04T09:46:16.7052984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7053129Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7053438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7053574Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7053902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.7054038Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7054386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7054487Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7054490Z 2025-12-04T09:46:16.7054619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7054847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7054918Z res = mod(**inputs) 2025-12-04T09:46:16.7055232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7055308Z outputs = self.mobilebert( 2025-12-04T09:46:16.7055628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7055737Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7056046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7056124Z layer_outputs = layer_module( 2025-12-04T09:46:16.7056454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.7056590Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.7056915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7057006Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7057010Z 2025-12-04T09:46:16.7057122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7057346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7057419Z res = mod(**inputs) 2025-12-04T09:46:16.7057734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7057811Z outputs = self.mobilebert( 2025-12-04T09:46:16.7058133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7058219Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7058590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7058667Z layer_outputs = layer_module( 2025-12-04T09:46:16.7058984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.7059117Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.7059438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7059564Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7059568Z 2025-12-04T09:46:16.7059679Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7059911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7059979Z res = mod(**inputs) 2025-12-04T09:46:16.7060321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7060398Z outputs = self.mobilebert( 2025-12-04T09:46:16.7060699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7060784Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7061095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7061220Z layer_outputs = layer_module( 2025-12-04T09:46:16.7061524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7061712Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7062026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.7062128Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.7062132Z 2025-12-04T09:46:16.7062240Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7062458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7062526Z res = mod(**inputs) 2025-12-04T09:46:16.7062852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7062929Z outputs = self.mobilebert( 2025-12-04T09:46:16.7063228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7063314Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7063617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7063700Z layer_outputs = layer_module( 2025-12-04T09:46:16.7063997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7064165Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7064470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.7064605Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.7064902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7065009Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7065013Z 2025-12-04T09:46:16.7065124Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7065340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7065410Z res = mod(**inputs) 2025-12-04T09:46:16.7065710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7065795Z outputs = self.mobilebert( 2025-12-04T09:46:16.7066102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7066192Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7066503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7066580Z layer_outputs = layer_module( 2025-12-04T09:46:16.7066915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7067097Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7067403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.7067536Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.7067832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.7067951Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7067955Z 2025-12-04T09:46:16.7068063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7068273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7068367Z res = mod(**inputs) 2025-12-04T09:46:16.7068669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7068751Z outputs = self.mobilebert( 2025-12-04T09:46:16.7069051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7069128Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7069432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7069531Z layer_outputs = layer_module( 2025-12-04T09:46:16.7069836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7070004Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7070309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.7070451Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.7070750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.7070881Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7071185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7071288Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7071292Z 2025-12-04T09:46:16.7071409Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7071619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7071689Z res = mod(**inputs) 2025-12-04T09:46:16.7071996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7072071Z outputs = self.mobilebert( 2025-12-04T09:46:16.7072378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7072456Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7072757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7072842Z layer_outputs = layer_module( 2025-12-04T09:46:16.7073141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.7073327Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.7073651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.7073775Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.7074093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.7074185Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.7074189Z 2025-12-04T09:46:16.7074303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7074527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7074618Z res = mod(**inputs) 2025-12-04T09:46:16.7074939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7075017Z outputs = self.mobilebert( 2025-12-04T09:46:16.7075347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7075435Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7075747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7075830Z layer_outputs = layer_module( 2025-12-04T09:46:16.7076138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7076268Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7076588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.7076668Z self_outputs = self.self( 2025-12-04T09:46:16.7076978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.7077065Z self.value(value_tensor) 2025-12-04T09:46:16.7077068Z 2025-12-04T09:46:16.7077183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7077406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7077477Z res = mod(**inputs) 2025-12-04T09:46:16.7077784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7077868Z outputs = self.mobilebert( 2025-12-04T09:46:16.7078178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7078267Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7078577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7078655Z layer_outputs = layer_module( 2025-12-04T09:46:16.7078973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.7079148Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.7079456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.7079585Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.7079897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.7079997Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.7080001Z 2025-12-04T09:46:16.7080115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7080407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7080497Z res = mod(**inputs) 2025-12-04T09:46:16.7080829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7080919Z outputs = self.mobilebert( 2025-12-04T09:46:16.7081232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7081314Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7081637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7081739Z layer_outputs = layer_module( 2025-12-04T09:46:16.7082050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.7082253Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.7082565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.7082695Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.7083000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.7083097Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.7083415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7083544Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7083548Z 2025-12-04T09:46:16.7083674Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7083896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7083967Z res = mod(**inputs) 2025-12-04T09:46:16.7084294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7084370Z outputs = self.mobilebert( 2025-12-04T09:46:16.7084688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7084767Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7085079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7085170Z layer_outputs = layer_module( 2025-12-04T09:46:16.7085485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7085578Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7085901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.7085979Z self_outputs = self.self( 2025-12-04T09:46:16.7086300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.7086378Z self.query(query_tensor) 2025-12-04T09:46:16.7086381Z 2025-12-04T09:46:16.7086494Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7086721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7086794Z res = mod(**inputs) 2025-12-04T09:46:16.7087114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7087191Z outputs = self.mobilebert( 2025-12-04T09:46:16.7087506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7087611Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7087922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7088001Z layer_outputs = layer_module( 2025-12-04T09:46:16.7088315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7088411Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7088727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.7088825Z self_outputs = self.self( 2025-12-04T09:46:16.7089145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.7089256Z self.key(key_tensor) 2025-12-04T09:46:16.7089260Z 2025-12-04T09:46:16.7089353Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.7089447Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.7089559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7089767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7089843Z res = mod(**inputs) 2025-12-04T09:46:16.7090144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7090246Z outputs = self.mobilebert( 2025-12-04T09:46:16.7090565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7090641Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7090958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7091034Z layer_outputs = layer_module( 2025-12-04T09:46:16.7091349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7091451Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7091772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.7091909Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.7092237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.7092333Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7092336Z 2025-12-04T09:46:16.7092455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7092676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7092747Z res = mod(**inputs) 2025-12-04T09:46:16.7093072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7093149Z outputs = self.mobilebert( 2025-12-04T09:46:16.7093473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7093551Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7093871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7093957Z layer_outputs = layer_module( 2025-12-04T09:46:16.7094274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7094368Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7094725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.7094862Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.7095192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.7095325Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7095637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7095765Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7095769Z 2025-12-04T09:46:16.7095877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7096111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7096183Z res = mod(**inputs) 2025-12-04T09:46:16.7096501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7096587Z outputs = self.mobilebert( 2025-12-04T09:46:16.7096893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7096980Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7097298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7097399Z layer_outputs = layer_module( 2025-12-04T09:46:16.7097720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7097829Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7098158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7098291Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7098607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7098708Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7098712Z 2025-12-04T09:46:16.7098824Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7099045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7099127Z res = mod(**inputs) 2025-12-04T09:46:16.7099444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7099540Z outputs = self.mobilebert( 2025-12-04T09:46:16.7099869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7099956Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7100275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7100352Z layer_outputs = layer_module( 2025-12-04T09:46:16.7100689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7100797Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7101124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7101247Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7101585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7101736Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7101741Z 2025-12-04T09:46:16.7101854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7102072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7102142Z res = mod(**inputs) 2025-12-04T09:46:16.7102458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7102565Z outputs = self.mobilebert( 2025-12-04T09:46:16.7102876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7102963Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7103315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7103397Z layer_outputs = layer_module( 2025-12-04T09:46:16.7103714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7103817Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7104135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7104284Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7104620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.7104722Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7104726Z 2025-12-04T09:46:16.7104840Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7105055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7105135Z res = mod(**inputs) 2025-12-04T09:46:16.7105446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7105531Z outputs = self.mobilebert( 2025-12-04T09:46:16.7105844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7105924Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7106245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7106324Z layer_outputs = layer_module( 2025-12-04T09:46:16.7106636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7106749Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7107060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7107205Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7107513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.7107647Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7107965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7108068Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7108071Z 2025-12-04T09:46:16.7108193Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7108412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7108483Z res = mod(**inputs) 2025-12-04T09:46:16.7108826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7108905Z outputs = self.mobilebert( 2025-12-04T09:46:16.7109221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7109307Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7109619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7109722Z layer_outputs = layer_module( 2025-12-04T09:46:16.7110031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7110155Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7110475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7110599Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7110915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7111007Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7111011Z 2025-12-04T09:46:16.7111122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7111367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7111438Z res = mod(**inputs) 2025-12-04T09:46:16.7111756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7111835Z outputs = self.mobilebert( 2025-12-04T09:46:16.7112142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7112227Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7112535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7112612Z layer_outputs = layer_module( 2025-12-04T09:46:16.7112925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7113029Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7113341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7113464Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7113775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7113908Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7113912Z 2025-12-04T09:46:16.7114025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7114251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7114322Z res = mod(**inputs) 2025-12-04T09:46:16.7114627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7114718Z outputs = self.mobilebert( 2025-12-04T09:46:16.7115025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7115105Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7115443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7115537Z layer_outputs = layer_module( 2025-12-04T09:46:16.7115855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7115957Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7116288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7116437Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7116764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.7116862Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7116866Z 2025-12-04T09:46:16.7116994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7117214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7117293Z res = mod(**inputs) 2025-12-04T09:46:16.7117600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7117677Z outputs = self.mobilebert( 2025-12-04T09:46:16.7118017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7118117Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7118433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7118509Z layer_outputs = layer_module( 2025-12-04T09:46:16.7118840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7118949Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7119268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7119412Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7119722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.7119855Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7120174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7120274Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7120278Z 2025-12-04T09:46:16.7120473Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7120700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7120772Z res = mod(**inputs) 2025-12-04T09:46:16.7121087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7121165Z outputs = self.mobilebert( 2025-12-04T09:46:16.7121483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7121573Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7121880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7121968Z layer_outputs = layer_module( 2025-12-04T09:46:16.7122277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7122380Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7122721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7122846Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7123161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7123255Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7123259Z 2025-12-04T09:46:16.7123374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7123624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7123695Z res = mod(**inputs) 2025-12-04T09:46:16.7124034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7124120Z outputs = self.mobilebert( 2025-12-04T09:46:16.7124441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7124527Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7124833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7124910Z layer_outputs = layer_module( 2025-12-04T09:46:16.7125222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7125345Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7125658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7125780Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7126089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7126219Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7126223Z 2025-12-04T09:46:16.7126335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7126558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7126628Z res = mod(**inputs) 2025-12-04T09:46:16.7126932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7127019Z outputs = self.mobilebert( 2025-12-04T09:46:16.7127325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7127404Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7127718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7127794Z layer_outputs = layer_module( 2025-12-04T09:46:16.7128106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7128207Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7128514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7128660Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7128969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.7129070Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7129073Z 2025-12-04T09:46:16.7129184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7129418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7129498Z res = mod(**inputs) 2025-12-04T09:46:16.7129805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7129881Z outputs = self.mobilebert( 2025-12-04T09:46:16.7130197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7130279Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7130625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7130704Z layer_outputs = layer_module( 2025-12-04T09:46:16.7131030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7131144Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7131451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7131594Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7131903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.7132060Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7132380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7132480Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7132483Z 2025-12-04T09:46:16.7132605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7132828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7132900Z res = mod(**inputs) 2025-12-04T09:46:16.7133217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7133293Z outputs = self.mobilebert( 2025-12-04T09:46:16.7133605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7133694Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7134005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7134091Z layer_outputs = layer_module( 2025-12-04T09:46:16.7134407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.7134542Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.7134862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7134955Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7134959Z 2025-12-04T09:46:16.7135079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7135298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7135368Z res = mod(**inputs) 2025-12-04T09:46:16.7135685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7135762Z outputs = self.mobilebert( 2025-12-04T09:46:16.7136084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7136172Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7136499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7136586Z layer_outputs = layer_module( 2025-12-04T09:46:16.7136891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.7137022Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.7137338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7137486Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7137491Z 2025-12-04T09:46:16.7137610Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7137847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7137921Z res = mod(**inputs) 2025-12-04T09:46:16.7138244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7138322Z outputs = self.mobilebert( 2025-12-04T09:46:16.7138630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7138718Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7139025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7139130Z layer_outputs = layer_module( 2025-12-04T09:46:16.7139439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7139615Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7139929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.7140033Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.7140037Z 2025-12-04T09:46:16.7140160Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7140374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7140444Z res = mod(**inputs) 2025-12-04T09:46:16.7140758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7140836Z outputs = self.mobilebert( 2025-12-04T09:46:16.7141141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7141229Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7141558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7141643Z layer_outputs = layer_module( 2025-12-04T09:46:16.7141949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7142120Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7142457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.7142596Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.7142910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7143011Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7143014Z 2025-12-04T09:46:16.7143144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7143371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7143443Z res = mod(**inputs) 2025-12-04T09:46:16.7143780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7143857Z outputs = self.mobilebert( 2025-12-04T09:46:16.7144184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7144289Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7144597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7144672Z layer_outputs = layer_module( 2025-12-04T09:46:16.7145028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7145200Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7145516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.7145651Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.7145958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.7146079Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7146083Z 2025-12-04T09:46:16.7146198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7146422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7146492Z res = mod(**inputs) 2025-12-04T09:46:16.7146799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7146885Z outputs = self.mobilebert( 2025-12-04T09:46:16.7147430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7147515Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7147833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7147916Z layer_outputs = layer_module( 2025-12-04T09:46:16.7148230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7148402Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7148712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.7148860Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.7149171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.7149314Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7149621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7149723Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7149728Z 2025-12-04T09:46:16.7149851Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7150069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7150149Z res = mod(**inputs) 2025-12-04T09:46:16.7150512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7150592Z outputs = self.mobilebert( 2025-12-04T09:46:16.7150911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7150991Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7151305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7151421Z layer_outputs = layer_module( 2025-12-04T09:46:16.7151727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.7151913Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.7152248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.7152371Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.7152686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.7152776Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.7152781Z 2025-12-04T09:46:16.7152904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7153125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7153224Z res = mod(**inputs) 2025-12-04T09:46:16.7153538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7153616Z outputs = self.mobilebert( 2025-12-04T09:46:16.7153928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7154018Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7154322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7154408Z layer_outputs = layer_module( 2025-12-04T09:46:16.7154713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7154809Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7155126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.7155203Z self_outputs = self.self( 2025-12-04T09:46:16.7155540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:46:16.7155619Z self.value(value_tensor) 2025-12-04T09:46:16.7155623Z 2025-12-04T09:46:16.7155739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7155964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7156033Z res = mod(**inputs) 2025-12-04T09:46:16.7156363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7156446Z outputs = self.mobilebert( 2025-12-04T09:46:16.7156754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7156844Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7157152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7157230Z layer_outputs = layer_module( 2025-12-04T09:46:16.7157572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.7157750Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.7158088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:46:16.7158210Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:46:16.7158523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:46:16.7158640Z layer_input = self.dense(hidden_states) 2025-12-04T09:46:16.7158645Z 2025-12-04T09:46:16.7158757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7158997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7159071Z res = mod(**inputs) 2025-12-04T09:46:16.7159375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7159461Z outputs = self.mobilebert( 2025-12-04T09:46:16.7159775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7159854Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7160190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7160288Z layer_outputs = layer_module( 2025-12-04T09:46:16.7160676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:46:16.7160862Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:46:16.7161200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:46:16.7161328Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:46:16.7161637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:46:16.7161739Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:46:16.7162116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7162223Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7162227Z 2025-12-04T09:46:16.7162346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7162565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7162635Z res = mod(**inputs) 2025-12-04T09:46:16.7162960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7163038Z outputs = self.mobilebert( 2025-12-04T09:46:16.7163358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7163439Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7163747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7163838Z layer_outputs = layer_module( 2025-12-04T09:46:16.7164146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7164250Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7164573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.7164670Z self_outputs = self.self( 2025-12-04T09:46:16.7164987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:46:16.7165067Z self.query(query_tensor) 2025-12-04T09:46:16.7165071Z 2025-12-04T09:46:16.7165183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7165409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7165501Z res = mod(**inputs) 2025-12-04T09:46:16.7165827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7165907Z outputs = self.mobilebert( 2025-12-04T09:46:16.7166242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7166334Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7166647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7166728Z layer_outputs = layer_module( 2025-12-04T09:46:16.7167026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7167118Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7167451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:46:16.7167531Z self_outputs = self.self( 2025-12-04T09:46:16.7167838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:46:16.7167923Z self.key(key_tensor) 2025-12-04T09:46:16.7167926Z 2025-12-04T09:46:16.7168015Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.7168111Z cudagraph partition due to non gpu ops 2025-12-04T09:46:16.7168225Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7168441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7168520Z res = mod(**inputs) 2025-12-04T09:46:16.7168826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7168906Z outputs = self.mobilebert( 2025-12-04T09:46:16.7169224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7169302Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7169623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7169701Z layer_outputs = layer_module( 2025-12-04T09:46:16.7170016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7170116Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7170414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.7170558Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.7170865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:46:16.7170961Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7170965Z 2025-12-04T09:46:16.7171084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7171302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7171437Z res = mod(**inputs) 2025-12-04T09:46:16.7171749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7171826Z outputs = self.mobilebert( 2025-12-04T09:46:16.7172141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7172239Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7172548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7172655Z layer_outputs = layer_module( 2025-12-04T09:46:16.7172966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:46:16.7173083Z self_attention_outputs = self.attention( 2025-12-04T09:46:16.7173392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:46:16.7173526Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:46:16.7173840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:46:16.7173978Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7174292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7174415Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7174418Z 2025-12-04T09:46:16.7174531Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7174758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7174829Z res = mod(**inputs) 2025-12-04T09:46:16.7175141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7175226Z outputs = self.mobilebert( 2025-12-04T09:46:16.7175538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7175624Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7175933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7176013Z layer_outputs = layer_module( 2025-12-04T09:46:16.7176334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7176440Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7176760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7176883Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7177196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7177298Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7177302Z 2025-12-04T09:46:16.7177414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7177641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7177713Z res = mod(**inputs) 2025-12-04T09:46:16.7178021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7178104Z outputs = self.mobilebert( 2025-12-04T09:46:16.7178448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7178529Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7178848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7178924Z layer_outputs = layer_module( 2025-12-04T09:46:16.7179251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7179356Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7179684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7179815Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7180149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7180285Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7180288Z 2025-12-04T09:46:16.7180401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7180619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7180696Z res = mod(**inputs) 2025-12-04T09:46:16.7181016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7181125Z outputs = self.mobilebert( 2025-12-04T09:46:16.7181442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7181520Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7181846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7181926Z layer_outputs = layer_module( 2025-12-04T09:46:16.7182234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7182349Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7182667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7182816Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7183125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.7183218Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7183221Z 2025-12-04T09:46:16.7183340Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7183557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7183625Z res = mod(**inputs) 2025-12-04T09:46:16.7183946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7184022Z outputs = self.mobilebert( 2025-12-04T09:46:16.7184348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7184426Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7184735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7184821Z layer_outputs = layer_module( 2025-12-04T09:46:16.7185129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7185239Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7185576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7185717Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7186047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.7186181Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7186494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7186622Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7186626Z 2025-12-04T09:46:16.7186739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7186985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7187059Z res = mod(**inputs) 2025-12-04T09:46:16.7187372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7187460Z outputs = self.mobilebert( 2025-12-04T09:46:16.7187769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7187857Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7188165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7188264Z layer_outputs = layer_module( 2025-12-04T09:46:16.7188583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7188685Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7189000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7189132Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7189446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7189547Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7189550Z 2025-12-04T09:46:16.7189661Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7189882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7189963Z res = mod(**inputs) 2025-12-04T09:46:16.7190276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7190362Z outputs = self.mobilebert( 2025-12-04T09:46:16.7190678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7190756Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7191080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7191156Z layer_outputs = layer_module( 2025-12-04T09:46:16.7191473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7191577Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7191887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7192016Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7192347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7192474Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7192486Z 2025-12-04T09:46:16.7192602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7192821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7192901Z res = mod(**inputs) 2025-12-04T09:46:16.7193210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7193310Z outputs = self.mobilebert( 2025-12-04T09:46:16.7193644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7193722Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7194066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7194149Z layer_outputs = layer_module( 2025-12-04T09:46:16.7194460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7194570Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7194879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7195042Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7195353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.7195445Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7195449Z 2025-12-04T09:46:16.7195568Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7195783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7195855Z res = mod(**inputs) 2025-12-04T09:46:16.7196167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7196243Z outputs = self.mobilebert( 2025-12-04T09:46:16.7196553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7196633Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7196939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7197025Z layer_outputs = layer_module( 2025-12-04T09:46:16.7197330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7197431Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7197744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7197879Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7198192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.7198325Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7198633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7198740Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7198743Z 2025-12-04T09:46:16.7198855Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7199078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7199165Z res = mod(**inputs) 2025-12-04T09:46:16.7199482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7199565Z outputs = self.mobilebert( 2025-12-04T09:46:16.7199877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7199963Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7200280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7200451Z layer_outputs = layer_module( 2025-12-04T09:46:16.7200781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7200905Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7201218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7201351Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7201657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7201759Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7201767Z 2025-12-04T09:46:16.7201900Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7202122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7202203Z res = mod(**inputs) 2025-12-04T09:46:16.7202521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7202609Z outputs = self.mobilebert( 2025-12-04T09:46:16.7202915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7202993Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7203304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7203382Z layer_outputs = layer_module( 2025-12-04T09:46:16.7203686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7203797Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7204104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:46:16.7204232Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:46:16.7204536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7204657Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7204661Z 2025-12-04T09:46:16.7204780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7204990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7205067Z res = mod(**inputs) 2025-12-04T09:46:16.7205372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7205449Z outputs = self.mobilebert( 2025-12-04T09:46:16.7205761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7205837Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7206162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7206248Z layer_outputs = layer_module( 2025-12-04T09:46:16.7206545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7206654Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7206951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7207087Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7207412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:46:16.7207502Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7207506Z 2025-12-04T09:46:16.7207649Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7207862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7207931Z res = mod(**inputs) 2025-12-04T09:46:16.7208237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7208311Z outputs = self.mobilebert( 2025-12-04T09:46:16.7208622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7208720Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7209028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7209109Z layer_outputs = layer_module( 2025-12-04T09:46:16.7209418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:46:16.7209517Z attention_output = ffn_module(attention_output) 2025-12-04T09:46:16.7209833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:46:16.7209965Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:46:16.7210277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:46:16.7210407Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7210716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7210822Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7210825Z 2025-12-04T09:46:16.7210936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7211156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7211225Z res = mod(**inputs) 2025-12-04T09:46:16.7211526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7211610Z outputs = self.mobilebert( 2025-12-04T09:46:16.7211912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7211990Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7212307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7212386Z layer_outputs = layer_module( 2025-12-04T09:46:16.7212707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.7212841Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.7213172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:46:16.7213276Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7213280Z 2025-12-04T09:46:16.7213396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7213626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7213697Z res = mod(**inputs) 2025-12-04T09:46:16.7214023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7214108Z outputs = self.mobilebert( 2025-12-04T09:46:16.7214431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7214512Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7214829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7214906Z layer_outputs = layer_module( 2025-12-04T09:46:16.7215221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:46:16.7215355Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:46:16.7215660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:46:16.7215813Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:46:16.7215817Z 2025-12-04T09:46:16.7215927Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7216152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7216223Z res = mod(**inputs) 2025-12-04T09:46:16.7216529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7216614Z outputs = self.mobilebert( 2025-12-04T09:46:16.7216920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7217005Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7217311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7217390Z layer_outputs = layer_module( 2025-12-04T09:46:16.7217705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7217882Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7218189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:46:16.7218300Z layer_output = self.dense(intermediate_states) 2025-12-04T09:46:16.7218304Z 2025-12-04T09:46:16.7218416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7218638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7218706Z res = mod(**inputs) 2025-12-04T09:46:16.7219009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7219097Z outputs = self.mobilebert( 2025-12-04T09:46:16.7219403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7219489Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7219821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7219900Z layer_outputs = layer_module( 2025-12-04T09:46:16.7220212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7220384Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7220693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:46:16.7220858Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:46:16.7221167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7221293Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7221297Z 2025-12-04T09:46:16.7221411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7221632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7221711Z res = mod(**inputs) 2025-12-04T09:46:16.7222017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7222101Z outputs = self.mobilebert( 2025-12-04T09:46:16.7222407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7222506Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7222821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7222898Z layer_outputs = layer_module( 2025-12-04T09:46:16.7223206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7223386Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7223694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.7223839Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.7224149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:46:16.7224245Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:46:16.7224256Z 2025-12-04T09:46:16.7224369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7224587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7224667Z res = mod(**inputs) 2025-12-04T09:46:16.7224977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:46:16.7225053Z outputs = self.mobilebert( 2025-12-04T09:46:16.7225368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:46:16.7225446Z encoder_outputs = self.encoder( 2025-12-04T09:46:16.7225761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:46:16.7225841Z layer_outputs = layer_module( 2025-12-04T09:46:16.7226145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:46:16.7226321Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:46:16.7226647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:46:16.7226784Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:46:16.7227095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:46:16.7227230Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:46:16.7227548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:46:16.7227670Z return input_tensor * self.weight + self.bias 2025-12-04T09:46:16.7227674Z 2025-12-04T09:46:16.7227785Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7228009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7228096Z res = mod(**inputs) 2025-12-04T09:46:16.7228416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T09:46:16.7228521Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:46:16.7228832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T09:46:16.7228968Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:46:16.7229278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 631, in forward 2025-12-04T09:46:16.7229410Z hidden_states = self.transform(hidden_states) 2025-12-04T09:46:16.7229723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 609, in forward 2025-12-04T09:46:16.7229816Z hidden_states = self.dense(hidden_states) 2025-12-04T09:46:16.7229821Z 2025-12-04T09:46:16.7229941Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7230158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7230228Z res = mod(**inputs) 2025-12-04T09:46:16.7230549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T09:46:16.7230647Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:46:16.7230964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T09:46:16.7231088Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:46:16.7231400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-12-04T09:46:16.7231639Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-12-04T09:46:16.7231643Z 2025-12-04T09:46:16.7231757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7231981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7232051Z res = mod(**inputs) 2025-12-04T09:46:16.7232362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T09:46:16.7232467Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:46:16.7232779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T09:46:16.7232910Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:46:16.7233218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 633, in forward 2025-12-04T09:46:16.7233306Z hidden_states += self.decoder.bias 2025-12-04T09:46:16.7233310Z 2025-12-04T09:46:16.7233450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:16.7233672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:16.7233742Z res = mod(**inputs) 2025-12-04T09:46:16.7234066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 994, in forward 2025-12-04T09:46:16.7234275Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:46:16.7234280Z 2025-12-04T09:46:30.7816797Z Compilation time (from dynamo_timed): 41.339205703 2025-12-04T09:46:30.7817114Z pass 2025-12-04T09:46:30.7817430Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:30.7818562Z TIMING: _recursive_pre_grad_passes:0.02514 _recursive_joint_graph_passes:1.49412 _recursive_post_grad_passes:0.20399 async_compile.wait:0.8476 code_gen:12.28461 inductor_compile:17.18406 backend_compile:29.96907 gc:0.00044 entire_frame_compile:41.33921 total_wall_time:41.33921 2025-12-04T09:46:30.7819566Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:30412 | FakeTensor.__torch_dispatch__:15340 | ProxyTorchDispatchMode.__torch_dispatch__:8821 2025-12-04T09:46:30.7820117Z Dynamo produced 1 graphs covering 1449 ops with 0 graph breaks (0 unique) 2025-12-04T09:46:34.3556591Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:46:34.3557962Z import pynvml # type: ignore[import] 2025-12-04T09:46:37.8416439Z 2025-12-04T09:46:39.4387788Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:46:39.4392159Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:46:39.4397774Z cpu eval OPTForCausalLM 2025-12-04T09:46:41.3696098Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:42.1976455Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:43.0329369Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:50.4410483Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4414151Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4415394Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4419137Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4419886Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4420323Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4420655Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4420990Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4421323Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4421591Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4422364Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4427193Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4431652Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4436161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4438005Z res = mod(**inputs) 2025-12-04T09:46:50.4441442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4445132Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4445687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4451633Z outputs = self.model.decoder( 2025-12-04T09:46:50.4455140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4460206Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4462331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4462849Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4463227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4463612Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4464018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4464606Z return func(*args, **kwargs) 2025-12-04T09:46:50.4464991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4465472Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4465900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4466281Z return func(*args, **kwargs) 2025-12-04T09:46:50.4466663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4467100Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4467275Z 2025-12-04T09:46:50.4467402Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4467780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4468168Z res = mod(**inputs) 2025-12-04T09:46:50.4468505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4468862Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4469255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4469650Z outputs = self.model.decoder( 2025-12-04T09:46:50.4470009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4470368Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4470755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4471205Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4471569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4471953Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4472359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4472750Z return func(*args, **kwargs) 2025-12-04T09:46:50.4473124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4473535Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4473942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4474303Z return func(*args, **kwargs) 2025-12-04T09:46:50.4474644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4475030Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4475175Z 2025-12-04T09:46:50.4475292Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4475653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4475986Z res = mod(**inputs) 2025-12-04T09:46:50.4476328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4476720Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4477118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4477524Z outputs = self.model.decoder( 2025-12-04T09:46:50.4477900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4478275Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4478666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4479091Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4479470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4479883Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4480302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4481007Z return func(*args, **kwargs) 2025-12-04T09:46:50.4481401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4481840Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4482299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4482785Z return func(*args, **kwargs) 2025-12-04T09:46:50.4483168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4483588Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4483751Z 2025-12-04T09:46:50.4483844Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4484080Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4484330Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4484726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4485078Z res = mod(**inputs) 2025-12-04T09:46:50.4485425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4485810Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4486216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4486626Z outputs = self.model.decoder( 2025-12-04T09:46:50.4486997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4487374Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4487776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4488192Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4488567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4488963Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4489371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4489764Z return func(*args, **kwargs) 2025-12-04T09:46:50.4490153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4490585Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4491018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4491409Z return func(*args, **kwargs) 2025-12-04T09:46:50.4491817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4492250Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4492724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4493246Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4493448Z 2025-12-04T09:46:50.4493561Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4493950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4494306Z res = mod(**inputs) 2025-12-04T09:46:50.4494660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4495039Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4495488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4495900Z outputs = self.model.decoder( 2025-12-04T09:46:50.4496274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4496649Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4497059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4497466Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4497859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4498217Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4498592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4498971Z return func(*args, **kwargs) 2025-12-04T09:46:50.4499342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4499750Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4500154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4500530Z return func(*args, **kwargs) 2025-12-04T09:46:50.4500898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.4501304Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.4501441Z 2025-12-04T09:46:50.4501552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4501903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4502220Z res = mod(**inputs) 2025-12-04T09:46:50.4502552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4502968Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4503366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4503771Z outputs = self.model.decoder( 2025-12-04T09:46:50.4504176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4504533Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4504904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4505286Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4505640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4506002Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4506403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4506785Z return func(*args, **kwargs) 2025-12-04T09:46:50.4507158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.4507544Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.4507694Z 2025-12-04T09:46:50.4507802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4508169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4508517Z res = mod(**inputs) 2025-12-04T09:46:50.4508844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4509206Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4509607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4509991Z outputs = self.model.decoder( 2025-12-04T09:46:50.4510364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4510745Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4511147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4511541Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4511916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4512287Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4512682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4513085Z return func(*args, **kwargs) 2025-12-04T09:46:50.4513472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.4513877Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.4514031Z 2025-12-04T09:46:50.4514136Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4514497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4514825Z res = mod(**inputs) 2025-12-04T09:46:50.4515148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4515511Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4515895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4516280Z outputs = self.model.decoder( 2025-12-04T09:46:50.4516625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4516996Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4517398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4517806Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4518180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4518586Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4519006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4519399Z return func(*args, **kwargs) 2025-12-04T09:46:50.4519790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.4520221Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.4520481Z 2025-12-04T09:46:50.4520641Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4521065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4521432Z res = mod(**inputs) 2025-12-04T09:46:50.4521786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4522163Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4522567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4522996Z outputs = self.model.decoder( 2025-12-04T09:46:50.4523369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4523740Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4524158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4524572Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4524959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4525376Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4525793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4526200Z return func(*args, **kwargs) 2025-12-04T09:46:50.4526587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4527045Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4527504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4527948Z return func(*args, **kwargs) 2025-12-04T09:46:50.4528336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4528782Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4528961Z 2025-12-04T09:46:50.4529082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4529461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4529810Z res = mod(**inputs) 2025-12-04T09:46:50.4530229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4530622Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4531019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4531425Z outputs = self.model.decoder( 2025-12-04T09:46:50.4531799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4532184Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4532588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4532996Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4533372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4533758Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4534167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4534575Z return func(*args, **kwargs) 2025-12-04T09:46:50.4534971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4535400Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4535855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4536261Z return func(*args, **kwargs) 2025-12-04T09:46:50.4536647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4537063Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4537217Z 2025-12-04T09:46:50.4537331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4537723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4538083Z res = mod(**inputs) 2025-12-04T09:46:50.4538438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4538822Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4539243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4539675Z outputs = self.model.decoder( 2025-12-04T09:46:50.4540049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4540425Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4540818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4541220Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4541595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4542000Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4542408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4542821Z return func(*args, **kwargs) 2025-12-04T09:46:50.4543231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4543659Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4544087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4544487Z return func(*args, **kwargs) 2025-12-04T09:46:50.4544888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4545323Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4545490Z 2025-12-04T09:46:50.4545584Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4545826Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4546087Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4546477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4546824Z res = mod(**inputs) 2025-12-04T09:46:50.4547418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4547806Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4548215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4548624Z outputs = self.model.decoder( 2025-12-04T09:46:50.4548994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4549384Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4549796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4550209Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4550590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4550986Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4551474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4551872Z return func(*args, **kwargs) 2025-12-04T09:46:50.4552267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4552700Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4553133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4553552Z return func(*args, **kwargs) 2025-12-04T09:46:50.4553931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4554343Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4554847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4555343Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4555544Z 2025-12-04T09:46:50.4555654Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4556028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4556360Z res = mod(**inputs) 2025-12-04T09:46:50.4556702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4557105Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4557509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4557916Z outputs = self.model.decoder( 2025-12-04T09:46:50.4558291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4558676Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4559068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4559520Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4559897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4560286Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4560763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4561176Z return func(*args, **kwargs) 2025-12-04T09:46:50.4561577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4562001Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4562409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4562791Z return func(*args, **kwargs) 2025-12-04T09:46:50.4563162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.4563553Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.4563707Z 2025-12-04T09:46:50.4563815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4564185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4564517Z res = mod(**inputs) 2025-12-04T09:46:50.4564844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4565210Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4565599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4566001Z outputs = self.model.decoder( 2025-12-04T09:46:50.4566364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4566721Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4567105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4567480Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4567850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4568239Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4568627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4569017Z return func(*args, **kwargs) 2025-12-04T09:46:50.4569700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.4570116Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.4570266Z 2025-12-04T09:46:50.4570378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4570757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4571089Z res = mod(**inputs) 2025-12-04T09:46:50.4571424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4571809Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4572198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4572588Z outputs = self.model.decoder( 2025-12-04T09:46:50.4572938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4573304Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4573693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4574080Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4574431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4574802Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4575193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4575577Z return func(*args, **kwargs) 2025-12-04T09:46:50.4575943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.4576382Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.4576534Z 2025-12-04T09:46:50.4576645Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4576999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4577327Z res = mod(**inputs) 2025-12-04T09:46:50.4577652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4578008Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4578375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4578758Z outputs = self.model.decoder( 2025-12-04T09:46:50.4579101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4579450Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4579828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4580203Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4580570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4580930Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4581314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4581690Z return func(*args, **kwargs) 2025-12-04T09:46:50.4582049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.4582447Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.4582592Z 2025-12-04T09:46:50.4582696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4583069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4583388Z res = mod(**inputs) 2025-12-04T09:46:50.4583723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4584079Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4584451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4584820Z outputs = self.model.decoder( 2025-12-04T09:46:50.4585161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4585526Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4585890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4586264Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4586615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4586977Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4587351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4587722Z return func(*args, **kwargs) 2025-12-04T09:46:50.4588080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4588476Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4588872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4589244Z return func(*args, **kwargs) 2025-12-04T09:46:50.4589604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4590007Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4590180Z 2025-12-04T09:46:50.4590285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4590638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4590958Z res = mod(**inputs) 2025-12-04T09:46:50.4591274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4591626Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4591998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4592372Z outputs = self.model.decoder( 2025-12-04T09:46:50.4592715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4593062Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4593431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4593799Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4594164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4594523Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4594894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4595267Z return func(*args, **kwargs) 2025-12-04T09:46:50.4595631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4596051Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4596448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4596865Z return func(*args, **kwargs) 2025-12-04T09:46:50.4597279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4597699Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4597848Z 2025-12-04T09:46:50.4597961Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4598349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4598703Z res = mod(**inputs) 2025-12-04T09:46:50.4599054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4599467Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4599887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4600314Z outputs = self.model.decoder( 2025-12-04T09:46:50.4600777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4601176Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4601603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4602003Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4602391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4602791Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4603215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4603614Z return func(*args, **kwargs) 2025-12-04T09:46:50.4604013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4604445Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4604878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4605281Z return func(*args, **kwargs) 2025-12-04T09:46:50.4605673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4606094Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4606246Z 2025-12-04T09:46:50.4606337Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4606579Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4606841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4607222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4607569Z res = mod(**inputs) 2025-12-04T09:46:50.4607896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4608255Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4608636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4609002Z outputs = self.model.decoder( 2025-12-04T09:46:50.4609336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4609675Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4610037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4610413Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4610788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4611141Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4611536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4611918Z return func(*args, **kwargs) 2025-12-04T09:46:50.4612289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4612688Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4613093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4613474Z return func(*args, **kwargs) 2025-12-04T09:46:50.4613835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4614266Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4614724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4615221Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4615411Z 2025-12-04T09:46:50.4615521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4615894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4616231Z res = mod(**inputs) 2025-12-04T09:46:50.4616564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4616926Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4617316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4617710Z outputs = self.model.decoder( 2025-12-04T09:46:50.4618059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4618413Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4618783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4619170Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4619521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4619890Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4620272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4620647Z return func(*args, **kwargs) 2025-12-04T09:46:50.4621020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4621431Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4621836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4622218Z return func(*args, **kwargs) 2025-12-04T09:46:50.4622595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.4622980Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.4623117Z 2025-12-04T09:46:50.4623227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4623574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4623896Z res = mod(**inputs) 2025-12-04T09:46:50.4624231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4624610Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4624997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4625381Z outputs = self.model.decoder( 2025-12-04T09:46:50.4625750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4626107Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4626490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4626876Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4627231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4627618Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4628821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4629202Z return func(*args, **kwargs) 2025-12-04T09:46:50.4629562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.4629952Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.4630092Z 2025-12-04T09:46:50.4630207Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4630572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4630894Z res = mod(**inputs) 2025-12-04T09:46:50.4631226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4631584Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4631965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4632354Z outputs = self.model.decoder( 2025-12-04T09:46:50.4632706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4633070Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4633446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4633828Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4634186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4634552Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4634943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4635321Z return func(*args, **kwargs) 2025-12-04T09:46:50.4635689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.4636094Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.4636259Z 2025-12-04T09:46:50.4636367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4636730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4637066Z res = mod(**inputs) 2025-12-04T09:46:50.4637452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4637837Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4638239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4638640Z outputs = self.model.decoder( 2025-12-04T09:46:50.4639016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4639398Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4639821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4640234Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4640750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4641165Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4641596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4642020Z return func(*args, **kwargs) 2025-12-04T09:46:50.4642423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.4642843Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.4642994Z 2025-12-04T09:46:50.4643129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4643524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4643880Z res = mod(**inputs) 2025-12-04T09:46:50.4644230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4644617Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4645024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4645430Z outputs = self.model.decoder( 2025-12-04T09:46:50.4645795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4646180Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4646585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4646988Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4647542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4647939Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4648358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4648755Z return func(*args, **kwargs) 2025-12-04T09:46:50.4649154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T09:46:50.4649632Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T09:46:50.4649836Z 2025-12-04T09:46:50.4649956Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4650347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4650700Z res = mod(**inputs) 2025-12-04T09:46:50.4651052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4651434Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4651842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4652250Z outputs = self.model.decoder( 2025-12-04T09:46:50.4652675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4653052Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4653453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4653865Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4654217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4654602Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4655008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4655380Z return func(*args, **kwargs) 2025-12-04T09:46:50.4655768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4656175Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4656583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4656963Z return func(*args, **kwargs) 2025-12-04T09:46:50.4657325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4657749Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4657918Z 2025-12-04T09:46:50.4658055Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4658418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4658752Z res = mod(**inputs) 2025-12-04T09:46:50.4659092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4659454Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4659831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4660204Z outputs = self.model.decoder( 2025-12-04T09:46:50.4660546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4660888Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4661261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4661636Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4661984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4662341Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4662721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4663089Z return func(*args, **kwargs) 2025-12-04T09:46:50.4663443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4663842Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4664249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4664627Z return func(*args, **kwargs) 2025-12-04T09:46:50.4664984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4665380Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4665520Z 2025-12-04T09:46:50.4665636Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4666018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4666358Z res = mod(**inputs) 2025-12-04T09:46:50.4666748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4667115Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4667494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4667904Z outputs = self.model.decoder( 2025-12-04T09:46:50.4668276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4668662Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4669077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4669478Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4669877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4670273Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4670691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4671095Z return func(*args, **kwargs) 2025-12-04T09:46:50.4671486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4671914Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4672346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4672774Z return func(*args, **kwargs) 2025-12-04T09:46:50.4673169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4673584Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4673744Z 2025-12-04T09:46:50.4673834Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4674075Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4674325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4674712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4675065Z res = mod(**inputs) 2025-12-04T09:46:50.4675420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4675800Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4676211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4676625Z outputs = self.model.decoder( 2025-12-04T09:46:50.4676991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4677381Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4677791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4678203Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4678577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4678976Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4679387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4679787Z return func(*args, **kwargs) 2025-12-04T09:46:50.4680188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4680688Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4681151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4681575Z return func(*args, **kwargs) 2025-12-04T09:46:50.4682023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4682468Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4682958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4683488Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4683702Z 2025-12-04T09:46:50.4683816Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4684225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4684570Z res = mod(**inputs) 2025-12-04T09:46:50.4684943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4685332Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4685716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4686092Z outputs = self.model.decoder( 2025-12-04T09:46:50.4686446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4686808Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4687180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4687586Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4687963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4688356Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4688761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4689173Z return func(*args, **kwargs) 2025-12-04T09:46:50.4689589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4690004Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4690412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4690788Z return func(*args, **kwargs) 2025-12-04T09:46:50.4691165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.4691565Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.4691710Z 2025-12-04T09:46:50.4691815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4692171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4692492Z res = mod(**inputs) 2025-12-04T09:46:50.4692807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4693158Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4693553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4693953Z outputs = self.model.decoder( 2025-12-04T09:46:50.4694332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4694771Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4695159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4695539Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4695902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4696295Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4696683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4697067Z return func(*args, **kwargs) 2025-12-04T09:46:50.4697445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.4697838Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.4697978Z 2025-12-04T09:46:50.4698084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4698473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4698805Z res = mod(**inputs) 2025-12-04T09:46:50.4699155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4699508Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4699892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4700277Z outputs = self.model.decoder( 2025-12-04T09:46:50.4700619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4700979Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4701364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4701783Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4702158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4702556Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4702964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4703360Z return func(*args, **kwargs) 2025-12-04T09:46:50.4703753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.4704159Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.4704313Z 2025-12-04T09:46:50.4704427Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4704782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4705115Z res = mod(**inputs) 2025-12-04T09:46:50.4705450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4705808Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4706191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4706576Z outputs = self.model.decoder( 2025-12-04T09:46:50.4706929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4707281Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4707659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4708043Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4708403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4708767Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4709154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4709543Z return func(*args, **kwargs) 2025-12-04T09:46:50.4709903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.4710293Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.4710460Z 2025-12-04T09:46:50.4710570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4710954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4711302Z res = mod(**inputs) 2025-12-04T09:46:50.4711656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4712024Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4712409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4712811Z outputs = self.model.decoder( 2025-12-04T09:46:50.4713163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4713576Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4713956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4714343Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4714714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4715106Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4715507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4715926Z return func(*args, **kwargs) 2025-12-04T09:46:50.4716302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4716729Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4717169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4717574Z return func(*args, **kwargs) 2025-12-04T09:46:50.4717965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4718407Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4718594Z 2025-12-04T09:46:50.4718708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4719103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4719448Z res = mod(**inputs) 2025-12-04T09:46:50.4719805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4720196Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4720710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4721143Z outputs = self.model.decoder( 2025-12-04T09:46:50.4721540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4721945Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4722357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4722766Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4723150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4723545Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4723950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4724355Z return func(*args, **kwargs) 2025-12-04T09:46:50.4724749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4725186Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4725634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4726035Z return func(*args, **kwargs) 2025-12-04T09:46:50.4726427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4726836Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4726991Z 2025-12-04T09:46:50.4727108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4727515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4727861Z res = mod(**inputs) 2025-12-04T09:46:50.4728208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4728605Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4729018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4729424Z outputs = self.model.decoder( 2025-12-04T09:46:50.4729807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4730201Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4730611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4731029Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4731408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4731796Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4732202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4732597Z return func(*args, **kwargs) 2025-12-04T09:46:50.4732988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4733417Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4733837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4734236Z return func(*args, **kwargs) 2025-12-04T09:46:50.4734628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4735044Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4735194Z 2025-12-04T09:46:50.4735284Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4735516Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4735769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4736147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4736497Z res = mod(**inputs) 2025-12-04T09:46:50.4736848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4737225Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4737620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4738022Z outputs = self.model.decoder( 2025-12-04T09:46:50.4738393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4738766Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4739170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4739577Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4739976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4740364Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4740776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4741182Z return func(*args, **kwargs) 2025-12-04T09:46:50.4741569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4742006Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4742467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4742874Z return func(*args, **kwargs) 2025-12-04T09:46:50.4743286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4743791Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4744284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4744807Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4745004Z 2025-12-04T09:46:50.4745122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4745519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4745902Z res = mod(**inputs) 2025-12-04T09:46:50.4746250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4746632Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4747032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4747551Z outputs = self.model.decoder( 2025-12-04T09:46:50.4747910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4748285Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4748662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4749039Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4749386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4749755Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4750146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4750516Z return func(*args, **kwargs) 2025-12-04T09:46:50.4750882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4751282Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4751679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4752043Z return func(*args, **kwargs) 2025-12-04T09:46:50.4752403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.4752786Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.4752926Z 2025-12-04T09:46:50.4753030Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4753394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4753726Z res = mod(**inputs) 2025-12-04T09:46:50.4754047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4754390Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4754801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4755175Z outputs = self.model.decoder( 2025-12-04T09:46:50.4755525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4755891Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4756279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4756694Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4757065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4757456Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4757894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4758302Z return func(*args, **kwargs) 2025-12-04T09:46:50.4758689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.4759101Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.4759249Z 2025-12-04T09:46:50.4759368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4759752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4760189Z res = mod(**inputs) 2025-12-04T09:46:50.4760612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4761020Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4761454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4761874Z outputs = self.model.decoder( 2025-12-04T09:46:50.4762267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4762645Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4763077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4763498Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4763894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4764302Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4764735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4765156Z return func(*args, **kwargs) 2025-12-04T09:46:50.4765562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.4766006Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.4766188Z 2025-12-04T09:46:50.4766303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4766702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4767074Z res = mod(**inputs) 2025-12-04T09:46:50.4767435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4767830Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4768249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4768668Z outputs = self.model.decoder( 2025-12-04T09:46:50.4769057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4769465Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4769896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4770319Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4770707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4771111Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4771524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4771938Z return func(*args, **kwargs) 2025-12-04T09:46:50.4772367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.4772803Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.4772965Z 2025-12-04T09:46:50.4773112Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4773510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4773886Z res = mod(**inputs) 2025-12-04T09:46:50.4774242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4774634Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4775051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4775475Z outputs = self.model.decoder( 2025-12-04T09:46:50.4775882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4776264Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4776674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4777051Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4777411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4777781Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4778169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4778543Z return func(*args, **kwargs) 2025-12-04T09:46:50.4778911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T09:46:50.4779376Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T09:46:50.4779580Z 2025-12-04T09:46:50.4779694Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4780073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4780404Z res = mod(**inputs) 2025-12-04T09:46:50.4780735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4781093Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4781474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4781861Z outputs = self.model.decoder( 2025-12-04T09:46:50.4782213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4782570Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4782953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4783338Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4783689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4784063Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4784489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4784871Z return func(*args, **kwargs) 2025-12-04T09:46:50.4785239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4785647Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4786057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4786431Z return func(*args, **kwargs) 2025-12-04T09:46:50.4786822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4787246Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4787415Z 2025-12-04T09:46:50.4787547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4787920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4788251Z res = mod(**inputs) 2025-12-04T09:46:50.4788581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4788937Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4789310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4789694Z outputs = self.model.decoder( 2025-12-04T09:46:50.4790067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4790414Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4790790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4791177Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4791540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4791911Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4792304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4792691Z return func(*args, **kwargs) 2025-12-04T09:46:50.4793059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4793475Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4793891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4794295Z return func(*args, **kwargs) 2025-12-04T09:46:50.4794683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4795103Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4795254Z 2025-12-04T09:46:50.4795380Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4795765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4796122Z res = mod(**inputs) 2025-12-04T09:46:50.4796479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4796870Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4797278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4797695Z outputs = self.model.decoder( 2025-12-04T09:46:50.4798074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4798461Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4798885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4799289Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4799669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4800051Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4800543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4800968Z return func(*args, **kwargs) 2025-12-04T09:46:50.4801393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4801825Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4802285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4802695Z return func(*args, **kwargs) 2025-12-04T09:46:50.4803086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4803516Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4803681Z 2025-12-04T09:46:50.4803776Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4804022Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4804283Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4804716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4805068Z res = mod(**inputs) 2025-12-04T09:46:50.4805415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4805803Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4806224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4806640Z outputs = self.model.decoder( 2025-12-04T09:46:50.4807011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4807394Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4807804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4808205Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4808590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4808985Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4809390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4809790Z return func(*args, **kwargs) 2025-12-04T09:46:50.4810189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4810626Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4811061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4811458Z return func(*args, **kwargs) 2025-12-04T09:46:50.4811852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4812288Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4812767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4813295Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4813500Z 2025-12-04T09:46:50.4813615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4814025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4814376Z res = mod(**inputs) 2025-12-04T09:46:50.4814731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4815119Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4815530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4815938Z outputs = self.model.decoder( 2025-12-04T09:46:50.4816330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4816715Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4817129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4817536Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4817921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4818403Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4818815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4819228Z return func(*args, **kwargs) 2025-12-04T09:46:50.4819630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4820087Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4820527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4820937Z return func(*args, **kwargs) 2025-12-04T09:46:50.4821335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.4821747Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.4821905Z 2025-12-04T09:46:50.4822020Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4822410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4822756Z res = mod(**inputs) 2025-12-04T09:46:50.4823100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4823488Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4823891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4824285Z outputs = self.model.decoder( 2025-12-04T09:46:50.4824660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4825041Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4825445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4825840Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4826218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4826611Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4827007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4827414Z return func(*args, **kwargs) 2025-12-04T09:46:50.4827803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.4828220Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.4828359Z 2025-12-04T09:46:50.4828468Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4828851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4829181Z res = mod(**inputs) 2025-12-04T09:46:50.4829506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4829869Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4830250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4830632Z outputs = self.model.decoder( 2025-12-04T09:46:50.4830979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4831385Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4831789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4832244Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4832639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4833037Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4833450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4833848Z return func(*args, **kwargs) 2025-12-04T09:46:50.4834240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.4834697Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.4834862Z 2025-12-04T09:46:50.4834980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4835356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4835701Z res = mod(**inputs) 2025-12-04T09:46:50.4836051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4836430Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4836836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4837239Z outputs = self.model.decoder( 2025-12-04T09:46:50.4837612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4837985Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4838388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4838796Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4839165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4839556Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4839967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4840458Z return func(*args, **kwargs) 2025-12-04T09:46:50.4840869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.4841305Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.4841459Z 2025-12-04T09:46:50.4841588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4842007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4842356Z res = mod(**inputs) 2025-12-04T09:46:50.4842715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4843115Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4843499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4843921Z outputs = self.model.decoder( 2025-12-04T09:46:50.4844277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4844641Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4845016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4845400Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4845762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4846148Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4846543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4846950Z return func(*args, **kwargs) 2025-12-04T09:46:50.4847440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4847852Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4848268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4848654Z return func(*args, **kwargs) 2025-12-04T09:46:50.4848902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4849073Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4849078Z 2025-12-04T09:46:50.4849183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4849388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4849462Z res = mod(**inputs) 2025-12-04T09:46:50.4849689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4849769Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4850025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4850103Z outputs = self.model.decoder( 2025-12-04T09:46:50.4850333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4850410Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4850659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4850745Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4850974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4851059Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4851314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4851385Z return func(*args, **kwargs) 2025-12-04T09:46:50.4851640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4851744Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4851988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4852073Z return func(*args, **kwargs) 2025-12-04T09:46:50.4852319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4852409Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4852413Z 2025-12-04T09:46:50.4852522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4852754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4852832Z res = mod(**inputs) 2025-12-04T09:46:50.4853049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4853124Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4853376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4853452Z outputs = self.model.decoder( 2025-12-04T09:46:50.4853680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4853786Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4854029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4854137Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4854367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4854459Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4854702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4854775Z return func(*args, **kwargs) 2025-12-04T09:46:50.4855027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4855145Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4855391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4855472Z return func(*args, **kwargs) 2025-12-04T09:46:50.4855717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4855811Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4855816Z 2025-12-04T09:46:50.4855898Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4855981Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4856096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4856299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4856367Z res = mod(**inputs) 2025-12-04T09:46:50.4856600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4856680Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4856944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4857023Z outputs = self.model.decoder( 2025-12-04T09:46:50.4857259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4857348Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4857605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4857683Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4857934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4858019Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4858287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4858364Z return func(*args, **kwargs) 2025-12-04T09:46:50.4858625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4858741Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4859020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4859104Z return func(*args, **kwargs) 2025-12-04T09:46:50.4859367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4859468Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4859771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4859914Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4859937Z 2025-12-04T09:46:50.4860049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4860269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4860356Z res = mod(**inputs) 2025-12-04T09:46:50.4860604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4860685Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4860942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4861028Z outputs = self.model.decoder( 2025-12-04T09:46:50.4861261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4861346Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4861625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4861705Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4861946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4862033Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4862294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4862374Z return func(*args, **kwargs) 2025-12-04T09:46:50.4862631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4862743Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4862996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4863072Z return func(*args, **kwargs) 2025-12-04T09:46:50.4863333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.4863420Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.4863425Z 2025-12-04T09:46:50.4863535Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4863753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4863824Z res = mod(**inputs) 2025-12-04T09:46:50.4864064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4864143Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4864401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4864489Z outputs = self.model.decoder( 2025-12-04T09:46:50.4864724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4864810Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4865068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4865146Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4865422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4865511Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4865773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4865857Z return func(*args, **kwargs) 2025-12-04T09:46:50.4866116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.4866213Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.4866247Z 2025-12-04T09:46:50.4866360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4866574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4866652Z res = mod(**inputs) 2025-12-04T09:46:50.4866905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4866987Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4867251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4867329Z outputs = self.model.decoder( 2025-12-04T09:46:50.4867570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4867648Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4867923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4868012Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4868251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4868344Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4868605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4868676Z return func(*args, **kwargs) 2025-12-04T09:46:50.4868923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.4869020Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.4869023Z 2025-12-04T09:46:50.4869128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4869341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4869411Z res = mod(**inputs) 2025-12-04T09:46:50.4869640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4869717Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4869959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4870043Z outputs = self.model.decoder( 2025-12-04T09:46:50.4870263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4870337Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4870586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4870659Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4870892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4870973Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4871226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4871310Z return func(*args, **kwargs) 2025-12-04T09:46:50.4871589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.4871684Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.4871688Z 2025-12-04T09:46:50.4871801Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4872013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4872090Z res = mod(**inputs) 2025-12-04T09:46:50.4872324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4872426Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4872694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4872777Z outputs = self.model.decoder( 2025-12-04T09:46:50.4873041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4873122Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4873367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4873450Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4873675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4873756Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4874021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4874093Z return func(*args, **kwargs) 2025-12-04T09:46:50.4874340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T09:46:50.4874477Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T09:46:50.4874481Z 2025-12-04T09:46:50.4874586Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4874791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4874856Z res = mod(**inputs) 2025-12-04T09:46:50.4875081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4875155Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4875396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4875480Z outputs = self.model.decoder( 2025-12-04T09:46:50.4875701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4875775Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4876027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4876102Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4876331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4876410Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4876653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4876734Z return func(*args, **kwargs) 2025-12-04T09:46:50.4876990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4877098Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4877361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4877437Z return func(*args, **kwargs) 2025-12-04T09:46:50.4877724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4877850Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4877854Z 2025-12-04T09:46:50.4877965Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4878184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4878252Z res = mod(**inputs) 2025-12-04T09:46:50.4878493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4878596Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4878857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4878943Z outputs = self.model.decoder( 2025-12-04T09:46:50.4879196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4879280Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4879544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4879622Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4879866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4879951Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4880231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4880316Z return func(*args, **kwargs) 2025-12-04T09:46:50.4880653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4880775Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4881040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4881118Z return func(*args, **kwargs) 2025-12-04T09:46:50.4881394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4881482Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4881487Z 2025-12-04T09:46:50.4881609Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4881832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4881902Z res = mod(**inputs) 2025-12-04T09:46:50.4882142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4882224Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4882482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4882571Z outputs = self.model.decoder( 2025-12-04T09:46:50.4882810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4882885Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4883135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4883211Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4883446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4883530Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4883776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4883859Z return func(*args, **kwargs) 2025-12-04T09:46:50.4884118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4884228Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4884473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4884542Z return func(*args, **kwargs) 2025-12-04T09:46:50.4884799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4884884Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4884901Z 2025-12-04T09:46:50.4884985Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4885073Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4885175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4885392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4885458Z res = mod(**inputs) 2025-12-04T09:46:50.4885673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4885754Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4885988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4886061Z outputs = self.model.decoder( 2025-12-04T09:46:50.4886281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4886386Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4886627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4886700Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4886919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4887010Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4887247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4887314Z return func(*args, **kwargs) 2025-12-04T09:46:50.4887557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4887655Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4887900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4887970Z return func(*args, **kwargs) 2025-12-04T09:46:50.4888204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4888313Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4888605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4888743Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4888747Z 2025-12-04T09:46:50.4888847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4889041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4889111Z res = mod(**inputs) 2025-12-04T09:46:50.4889329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4889403Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4889645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4889719Z outputs = self.model.decoder( 2025-12-04T09:46:50.4889955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4890030Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4890266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4890345Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4890562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4890647Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4890883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4891001Z return func(*args, **kwargs) 2025-12-04T09:46:50.4891245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4891357Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4891593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4891670Z return func(*args, **kwargs) 2025-12-04T09:46:50.4891925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.4892011Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.4892014Z 2025-12-04T09:46:50.4892115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4892327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4892399Z res = mod(**inputs) 2025-12-04T09:46:50.4892612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4892687Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4892931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4893003Z outputs = self.model.decoder( 2025-12-04T09:46:50.4893232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4893306Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4893542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4893619Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4893842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4893929Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4894165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4894238Z return func(*args, **kwargs) 2025-12-04T09:46:50.4894482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.4894561Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.4894564Z 2025-12-04T09:46:50.4894667Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4894867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4894931Z res = mod(**inputs) 2025-12-04T09:46:50.4895152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4895227Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4895467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4895546Z outputs = self.model.decoder( 2025-12-04T09:46:50.4895761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4895847Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4896091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4896162Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4896385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4896464Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4896701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4896795Z return func(*args, **kwargs) 2025-12-04T09:46:50.4897033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.4897151Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.4897155Z 2025-12-04T09:46:50.4897262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4897459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4897530Z res = mod(**inputs) 2025-12-04T09:46:50.4897751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4897825Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4898071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4898161Z outputs = self.model.decoder( 2025-12-04T09:46:50.4898388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4898460Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4898709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4898793Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4899028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4899107Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4899364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4899433Z return func(*args, **kwargs) 2025-12-04T09:46:50.4899691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.4899774Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.4899777Z 2025-12-04T09:46:50.4899883Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4900092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4900159Z res = mod(**inputs) 2025-12-04T09:46:50.4900395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4900471Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4900721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4900804Z outputs = self.model.decoder( 2025-12-04T09:46:50.4901028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4901107Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4901363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4901435Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4901672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4901770Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4902017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4902096Z return func(*args, **kwargs) 2025-12-04T09:46:50.4902341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4902443Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4902696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4902785Z return func(*args, **kwargs) 2025-12-04T09:46:50.4903034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4903163Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4903167Z 2025-12-04T09:46:50.4903274Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4903482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4903546Z res = mod(**inputs) 2025-12-04T09:46:50.4903774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4903851Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4904097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4904195Z outputs = self.model.decoder( 2025-12-04T09:46:50.4904421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4904498Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4904755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4904830Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4905066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4905145Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4905391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4905469Z return func(*args, **kwargs) 2025-12-04T09:46:50.4905718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4905827Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4906074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4906146Z return func(*args, **kwargs) 2025-12-04T09:46:50.4906403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4906484Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4906488Z 2025-12-04T09:46:50.4906593Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4906802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4906867Z res = mod(**inputs) 2025-12-04T09:46:50.4907098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4907176Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4907425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4907508Z outputs = self.model.decoder( 2025-12-04T09:46:50.4907734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4907823Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4908073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4908149Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4908379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4908458Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4908702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4908797Z return func(*args, **kwargs) 2025-12-04T09:46:50.4909049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4909179Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4909429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4909500Z return func(*args, **kwargs) 2025-12-04T09:46:50.4909753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4909842Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4909845Z 2025-12-04T09:46:50.4909929Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4910018Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4910142Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4910353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4910419Z res = mod(**inputs) 2025-12-04T09:46:50.4910643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4910726Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4910971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4911046Z outputs = self.model.decoder( 2025-12-04T09:46:50.4911276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4911350Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4911605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4911682Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4911907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4911995Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4912242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4912313Z return func(*args, **kwargs) 2025-12-04T09:46:50.4912564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4912662Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4912917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4912986Z return func(*args, **kwargs) 2025-12-04T09:46:50.4913233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4913341Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4913641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4913784Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4913788Z 2025-12-04T09:46:50.4913911Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4914115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4914189Z res = mod(**inputs) 2025-12-04T09:46:50.4914411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4914487Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4914741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4914832Z outputs = self.model.decoder( 2025-12-04T09:46:50.4915058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4915135Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4915396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4915480Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4915712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4915799Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4916042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4916112Z return func(*args, **kwargs) 2025-12-04T09:46:50.4916376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4916481Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4916738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4916820Z return func(*args, **kwargs) 2025-12-04T09:46:50.4917075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.4917169Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.4917173Z 2025-12-04T09:46:50.4917283Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4917492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4917568Z res = mod(**inputs) 2025-12-04T09:46:50.4917798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4917881Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4918143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4918219Z outputs = self.model.decoder( 2025-12-04T09:46:50.4918458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4918539Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4918795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4918879Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4919115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4919204Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4919461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4919536Z return func(*args, **kwargs) 2025-12-04T09:46:50.4919796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.4919882Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.4919886Z 2025-12-04T09:46:50.4920014Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4920237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4920307Z res = mod(**inputs) 2025-12-04T09:46:50.4920638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4920724Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4920989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4921106Z outputs = self.model.decoder( 2025-12-04T09:46:50.4921344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4921436Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4921718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4921799Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4922047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4922132Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4922392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4922475Z return func(*args, **kwargs) 2025-12-04T09:46:50.4922752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.4922867Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.4922872Z 2025-12-04T09:46:50.4922981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4923193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4923269Z res = mod(**inputs) 2025-12-04T09:46:50.4923504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4923585Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4923849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4923926Z outputs = self.model.decoder( 2025-12-04T09:46:50.4924164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4924246Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4924506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4924592Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4924829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4924916Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4925181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4925256Z return func(*args, **kwargs) 2025-12-04T09:46:50.4925520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.4925606Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.4925613Z 2025-12-04T09:46:50.4925722Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4925943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4926012Z res = mod(**inputs) 2025-12-04T09:46:50.4926255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4926334Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4926610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4926699Z outputs = self.model.decoder( 2025-12-04T09:46:50.4926935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4927014Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4927283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4927379Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4927625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4927707Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4927982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4928068Z return func(*args, **kwargs) 2025-12-04T09:46:50.4928325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T09:46:50.4928470Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T09:46:50.4928481Z 2025-12-04T09:46:50.4928592Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4928804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4928897Z res = mod(**inputs) 2025-12-04T09:46:50.4929132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4929212Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4929476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4929555Z outputs = self.model.decoder( 2025-12-04T09:46:50.4929794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4929872Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4930127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4930212Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4930449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4930533Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4930797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4930870Z return func(*args, **kwargs) 2025-12-04T09:46:50.4931137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4931245Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4931500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4931584Z return func(*args, **kwargs) 2025-12-04T09:46:50.4931840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4931970Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4931976Z 2025-12-04T09:46:50.4932087Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4932300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4932376Z res = mod(**inputs) 2025-12-04T09:46:50.4932610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4932689Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4932972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4933051Z outputs = self.model.decoder( 2025-12-04T09:46:50.4933289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4933368Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4933631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4933747Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4933993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4934079Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4934370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4934451Z return func(*args, **kwargs) 2025-12-04T09:46:50.4934728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4934838Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4935106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4935190Z return func(*args, **kwargs) 2025-12-04T09:46:50.4935484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4935578Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4935582Z 2025-12-04T09:46:50.4935693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4935907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4935983Z res = mod(**inputs) 2025-12-04T09:46:50.4936224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4936303Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4936571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4936648Z outputs = self.model.decoder( 2025-12-04T09:46:50.4936888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4936969Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4937228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4937312Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4937552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4937637Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4937911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4937984Z return func(*args, **kwargs) 2025-12-04T09:46:50.4938249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4938354Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4938616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4938700Z return func(*args, **kwargs) 2025-12-04T09:46:50.4938960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4939058Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4939062Z 2025-12-04T09:46:50.4939148Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4939254Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4939375Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4939589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4939658Z res = mod(**inputs) 2025-12-04T09:46:50.4939903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4939985Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4940269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4940349Z outputs = self.model.decoder( 2025-12-04T09:46:50.4940598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4940687Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4940946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4941025Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4941269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4941355Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4941624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4941721Z return func(*args, **kwargs) 2025-12-04T09:46:50.4941988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4942107Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4942373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4942452Z return func(*args, **kwargs) 2025-12-04T09:46:50.4942728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4942838Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4943176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4943315Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4943321Z 2025-12-04T09:46:50.4943434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4943657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4943726Z res = mod(**inputs) 2025-12-04T09:46:50.4943969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4944051Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4944312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4944400Z outputs = self.model.decoder( 2025-12-04T09:46:50.4944634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4944713Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4944976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4945058Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4945304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4945391Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4945654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4945755Z return func(*args, **kwargs) 2025-12-04T09:46:50.4946014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4946126Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4946385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4946458Z return func(*args, **kwargs) 2025-12-04T09:46:50.4946724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.4946829Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.4946833Z 2025-12-04T09:46:50.4946946Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4947420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4947498Z res = mod(**inputs) 2025-12-04T09:46:50.4947745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4947825Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4948082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4948171Z outputs = self.model.decoder( 2025-12-04T09:46:50.4948407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4948515Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4948781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4948860Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4949106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4949195Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4949450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4949537Z return func(*args, **kwargs) 2025-12-04T09:46:50.4949800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.4949895Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.4949901Z 2025-12-04T09:46:50.4950016Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4950231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4950312Z res = mod(**inputs) 2025-12-04T09:46:50.4950552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4950635Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4950904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4950985Z outputs = self.model.decoder( 2025-12-04T09:46:50.4951230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4951311Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4951572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4951665Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4951909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4951998Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4952268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4952374Z return func(*args, **kwargs) 2025-12-04T09:46:50.4952642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.4952747Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.4952751Z 2025-12-04T09:46:50.4952859Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4953076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4953147Z res = mod(**inputs) 2025-12-04T09:46:50.4953427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4953507Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4953778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4953864Z outputs = self.model.decoder( 2025-12-04T09:46:50.4954108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4954187Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4954460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4954538Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4954785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4954890Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4955150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4955234Z return func(*args, **kwargs) 2025-12-04T09:46:50.4955493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.4955581Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.4955591Z 2025-12-04T09:46:50.4955702Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4955912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4955987Z res = mod(**inputs) 2025-12-04T09:46:50.4956223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4956304Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4956573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4956650Z outputs = self.model.decoder( 2025-12-04T09:46:50.4956897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4956979Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4957244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4957333Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4957576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4957661Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4957946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4958027Z return func(*args, **kwargs) 2025-12-04T09:46:50.4958298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4958405Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4958682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4958794Z return func(*args, **kwargs) 2025-12-04T09:46:50.4959064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4959194Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4959199Z 2025-12-04T09:46:50.4959314Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4959533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4959615Z res = mod(**inputs) 2025-12-04T09:46:50.4959882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4959962Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4960254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4960335Z outputs = self.model.decoder( 2025-12-04T09:46:50.4960646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4960734Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4961003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4961092Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4961346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4961459Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4961743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4961830Z return func(*args, **kwargs) 2025-12-04T09:46:50.4962098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4962205Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4962467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4962548Z return func(*args, **kwargs) 2025-12-04T09:46:50.4962805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4962899Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4962905Z 2025-12-04T09:46:50.4963016Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4963230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4963308Z res = mod(**inputs) 2025-12-04T09:46:50.4963542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4963623Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4963890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4963967Z outputs = self.model.decoder( 2025-12-04T09:46:50.4964204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4964283Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4964539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4964626Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4964862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4964946Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4965212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4965306Z return func(*args, **kwargs) 2025-12-04T09:46:50.4965571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4965678Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4965935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4966017Z return func(*args, **kwargs) 2025-12-04T09:46:50.4966272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4966399Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4966403Z 2025-12-04T09:46:50.4966485Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4966566Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4966690Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4966895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4966963Z res = mod(**inputs) 2025-12-04T09:46:50.4967208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4967287Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4967554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4967650Z outputs = self.model.decoder( 2025-12-04T09:46:50.4967883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4967967Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4968223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4968301Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4968546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4968630Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4968926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4968997Z return func(*args, **kwargs) 2025-12-04T09:46:50.4969240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4969350Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4969596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4969666Z return func(*args, **kwargs) 2025-12-04T09:46:50.4969918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4970021Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4970337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4970477Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4970481Z 2025-12-04T09:46:50.4970591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4970807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4970878Z res = mod(**inputs) 2025-12-04T09:46:50.4971120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4971200Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4971462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4971550Z outputs = self.model.decoder( 2025-12-04T09:46:50.4971805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4971888Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4972158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4972236Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4972485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4972592Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4972855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4972935Z return func(*args, **kwargs) 2025-12-04T09:46:50.4973236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4973352Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4973612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4973687Z return func(*args, **kwargs) 2025-12-04T09:46:50.4973950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.4974040Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.4974061Z 2025-12-04T09:46:50.4974173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4974392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4974460Z res = mod(**inputs) 2025-12-04T09:46:50.4974703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4974782Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4975038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4975135Z outputs = self.model.decoder( 2025-12-04T09:46:50.4975368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4975445Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4975708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4975789Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4976033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4976117Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4976376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4976459Z return func(*args, **kwargs) 2025-12-04T09:46:50.4976716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.4976809Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.4976813Z 2025-12-04T09:46:50.4976922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4977135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4977215Z res = mod(**inputs) 2025-12-04T09:46:50.4977450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4977529Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4977797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4977875Z outputs = self.model.decoder( 2025-12-04T09:46:50.4978134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4978216Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4978471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4978558Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4978798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4978905Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4979170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4979244Z return func(*args, **kwargs) 2025-12-04T09:46:50.4979524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.4979631Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.4979635Z 2025-12-04T09:46:50.4979745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4979965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4980034Z res = mod(**inputs) 2025-12-04T09:46:50.4980271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4980379Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4980638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4980726Z outputs = self.model.decoder( 2025-12-04T09:46:50.4980961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4981039Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4981305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4981383Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4981625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4981710Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4981969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4982047Z return func(*args, **kwargs) 2025-12-04T09:46:50.4982291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.4982370Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.4982381Z 2025-12-04T09:46:50.4982493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4982711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4982787Z res = mod(**inputs) 2025-12-04T09:46:50.4983027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4983118Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4983381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4983459Z outputs = self.model.decoder( 2025-12-04T09:46:50.4983697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4983776Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4984038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4984123Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4984375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4984462Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4984728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4984802Z return func(*args, **kwargs) 2025-12-04T09:46:50.4985074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T09:46:50.4985229Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T09:46:50.4985233Z 2025-12-04T09:46:50.4985340Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4985551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4985634Z res = mod(**inputs) 2025-12-04T09:46:50.4985865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4985943Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4986188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4986273Z outputs = self.model.decoder( 2025-12-04T09:46:50.4986503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4986601Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4986879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4986958Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4987209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4987299Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4987559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4987642Z return func(*args, **kwargs) 2025-12-04T09:46:50.4987898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4988005Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4988272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4988350Z return func(*args, **kwargs) 2025-12-04T09:46:50.4988617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:46:50.4988738Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:50.4988742Z 2025-12-04T09:46:50.4988854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4989078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4989151Z res = mod(**inputs) 2025-12-04T09:46:50.4989394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4989477Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4989734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4989824Z outputs = self.model.decoder( 2025-12-04T09:46:50.4990061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4990139Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4990406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4990489Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4990794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4990878Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4991118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4991196Z return func(*args, **kwargs) 2025-12-04T09:46:50.4991438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4991563Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4991814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4991887Z return func(*args, **kwargs) 2025-12-04T09:46:50.4992166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:46:50.4992256Z key_states = self.k_proj(hidden_states) 2025-12-04T09:46:50.4992260Z 2025-12-04T09:46:50.4992369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4992586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4992655Z res = mod(**inputs) 2025-12-04T09:46:50.4992894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4992991Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4993249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4993333Z outputs = self.model.decoder( 2025-12-04T09:46:50.4993568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4993646Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4993910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4993989Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4994232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4994317Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4994570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4994654Z return func(*args, **kwargs) 2025-12-04T09:46:50.4994909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4995014Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4995275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4995345Z return func(*args, **kwargs) 2025-12-04T09:46:50.4995593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:46:50.4995677Z value_states = self.v_proj(hidden_states) 2025-12-04T09:46:50.4995681Z 2025-12-04T09:46:50.4995760Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4995849Z cudagraph partition due to non gpu ops 2025-12-04T09:46:50.4995950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.4996156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.4996224Z res = mod(**inputs) 2025-12-04T09:46:50.4996442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4996525Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4996781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.4996863Z outputs = self.model.decoder( 2025-12-04T09:46:50.4997105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.4997183Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.4997456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.4997529Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.4997773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.4997859Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.4998137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4998213Z return func(*args, **kwargs) 2025-12-04T09:46:50.4998478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.4998584Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.4998849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.4998922Z return func(*args, **kwargs) 2025-12-04T09:46:50.4999179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:46:50.4999313Z attn_output, attn_weights = attention_interface( 2025-12-04T09:46:50.4999630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:46:50.4999780Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:46:50.4999784Z 2025-12-04T09:46:50.4999894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.5000109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.5000188Z res = mod(**inputs) 2025-12-04T09:46:50.5000490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.5000575Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.5000843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.5000926Z outputs = self.model.decoder( 2025-12-04T09:46:50.5001173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.5001258Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.5001526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.5001618Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.5001874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.5001959Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.5002227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.5002302Z return func(*args, **kwargs) 2025-12-04T09:46:50.5002576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:46:50.5002686Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:50.5002944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.5003028Z return func(*args, **kwargs) 2025-12-04T09:46:50.5003290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:46:50.5003409Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:50.5003413Z 2025-12-04T09:46:50.5003527Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.5003739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.5003818Z res = mod(**inputs) 2025-12-04T09:46:50.5004051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.5004133Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.5004419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.5004497Z outputs = self.model.decoder( 2025-12-04T09:46:50.5004763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.5004844Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.5005110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.5005198Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.5005436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.5005519Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.5005789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.5005884Z return func(*args, **kwargs) 2025-12-04T09:46:50.5006155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:46:50.5006241Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:46:50.5006246Z 2025-12-04T09:46:50.5006357Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.5006580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.5006649Z res = mod(**inputs) 2025-12-04T09:46:50.5006898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.5006978Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.5007242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.5007332Z outputs = self.model.decoder( 2025-12-04T09:46:50.5007572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.5007650Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.5007923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.5008000Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.5008252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.5008337Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.5008603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.5008684Z return func(*args, **kwargs) 2025-12-04T09:46:50.5008946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:46:50.5009054Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:46:50.5009065Z 2025-12-04T09:46:50.5009176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.5009388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.5009466Z res = mod(**inputs) 2025-12-04T09:46:50.5009725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.5009809Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.5010072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:46:50.5010150Z outputs = self.model.decoder( 2025-12-04T09:46:50.5010389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.5010470Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.5010745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:46:50.5010836Z layer_outputs = decoder_layer( 2025-12-04T09:46:50.5011090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:50.5011176Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:50.5011442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:50.5011516Z return func(*args, **kwargs) 2025-12-04T09:46:50.5011782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:46:50.5011867Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:50.5011871Z 2025-12-04T09:46:50.5011982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.5012227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.5012295Z res = mod(**inputs) 2025-12-04T09:46:50.5012520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.5012596Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.5012842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 833, in forward 2025-12-04T09:46:50.5012949Z logits = self.lm_head(outputs[0]).contiguous() 2025-12-04T09:46:50.5012953Z 2025-12-04T09:46:50.5013061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:50.5013270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:50.5013346Z res = mod(**inputs) 2025-12-04T09:46:50.5013576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:46:50.5013665Z output = func(self, *args, **kwargs) 2025-12-04T09:46:50.5013920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 839, in forward 2025-12-04T09:46:50.5014002Z loss = self.loss_function( 2025-12-04T09:46:50.5014266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:46:50.5014452Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:46:50.5014730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:46:50.5014938Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:46:50.5014943Z 2025-12-04T09:47:02.0049535Z Compilation time (from dynamo_timed): 17.386597833 2025-12-04T09:47:02.0632123Z pass 2025-12-04T09:47:02.0632783Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:02.0633762Z TIMING: _recursive_pre_grad_passes:0.00782 _recursive_joint_graph_passes:0.6383 _recursive_post_grad_passes:0.08234 async_compile.wait:0.86695 code_gen:10.34497 inductor_compile:11.61786 backend_compile:14.76997 gc:0.00114 entire_frame_compile:17.3866 total_wall_time:17.3866 2025-12-04T09:47:02.0635373Z STATS: call_* op count: 379 | FakeTensorMode.__torch_dispatch__:7020 | FakeTensor.__torch_dispatch__:4035 | ProxyTorchDispatchMode.__torch_dispatch__:1957 2025-12-04T09:47:02.0635899Z Dynamo produced 1 graphs covering 379 ops with 0 graph breaks (0 unique) 2025-12-04T09:47:04.4184060Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:47:04.4190456Z import pynvml # type: ignore[import] 2025-12-04T09:47:07.7539571Z 2025-12-04T09:47:08.8040858Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:47:08.8041146Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:47:08.8055007Z cpu eval PLBartForCausalLM 2025-12-04T09:47:09.5452954Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:09.8491405Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:10.1507347Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:14.5869635Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.5876492Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.5878666Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.5878904Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.5879508Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.5879757Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.5880045Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5880635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5881008Z res = mod(**inputs) 2025-12-04T09:47:14.5881461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5881910Z outputs = self.model.decoder( 2025-12-04T09:47:14.5882325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5882742Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5883115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5883493Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5883901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5884440Z return func(*args, **kwargs) 2025-12-04T09:47:14.5884843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.5885281Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.5885693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5901376Z return func(*args, **kwargs) 2025-12-04T09:47:14.5901964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:47:14.5902467Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:14.5902678Z 2025-12-04T09:47:14.5902802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5903195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5903563Z res = mod(**inputs) 2025-12-04T09:47:14.5903950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5904373Z outputs = self.model.decoder( 2025-12-04T09:47:14.5904886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5905305Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5905681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5906072Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5906474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5906853Z return func(*args, **kwargs) 2025-12-04T09:47:14.5907280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.5907685Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.5908124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5908499Z return func(*args, **kwargs) 2025-12-04T09:47:14.5908882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:47:14.5909291Z key_states = self.k_proj(current_states) 2025-12-04T09:47:14.5909429Z 2025-12-04T09:47:14.5909548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5909913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5910257Z res = mod(**inputs) 2025-12-04T09:47:14.5910618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5911005Z outputs = self.model.decoder( 2025-12-04T09:47:14.5911374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5911756Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5912109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5912469Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5912845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5913217Z return func(*args, **kwargs) 2025-12-04T09:47:14.5913602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.5914003Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.5914388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5914747Z return func(*args, **kwargs) 2025-12-04T09:47:14.5915118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:47:14.5915524Z value_states = self.v_proj(current_states) 2025-12-04T09:47:14.5915672Z 2025-12-04T09:47:14.5915757Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.5915995Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5916342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5916659Z res = mod(**inputs) 2025-12-04T09:47:14.5917026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5917414Z outputs = self.model.decoder( 2025-12-04T09:47:14.5917798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5918186Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5918535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5918903Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5919288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5919665Z return func(*args, **kwargs) 2025-12-04T09:47:14.5920048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.5920566Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.5920990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5921397Z return func(*args, **kwargs) 2025-12-04T09:47:14.5921780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:47:14.5922229Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:14.5922708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:14.5923243Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:14.5923447Z 2025-12-04T09:47:14.5923564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5923961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5924318Z res = mod(**inputs) 2025-12-04T09:47:14.5924731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5925130Z outputs = self.model.decoder( 2025-12-04T09:47:14.5925527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5925938Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5926279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5926642Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5927021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5927391Z return func(*args, **kwargs) 2025-12-04T09:47:14.5927768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.5928193Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.5928600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5928965Z return func(*args, **kwargs) 2025-12-04T09:47:14.5929353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:47:14.5929767Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:14.5929905Z 2025-12-04T09:47:14.5930028Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5930377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5930696Z res = mod(**inputs) 2025-12-04T09:47:14.5931072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5931477Z outputs = self.model.decoder( 2025-12-04T09:47:14.5931868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5932273Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5932634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5932995Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5933403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5933778Z return func(*args, **kwargs) 2025-12-04T09:47:14.5934153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.5934582Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.5934761Z 2025-12-04T09:47:14.5934864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5935249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5935581Z res = mod(**inputs) 2025-12-04T09:47:14.5935952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5936374Z outputs = self.model.decoder( 2025-12-04T09:47:14.5936776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5937176Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5937536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5937908Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5938299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5938689Z return func(*args, **kwargs) 2025-12-04T09:47:14.5939075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.5939524Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.5939918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:14.5940274Z return self.act(input) 2025-12-04T09:47:14.5940397Z 2025-12-04T09:47:14.5940503Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5940869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5941190Z res = mod(**inputs) 2025-12-04T09:47:14.5941566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5941965Z outputs = self.model.decoder( 2025-12-04T09:47:14.5942354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5942756Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5943112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5943482Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5943863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5944244Z return func(*args, **kwargs) 2025-12-04T09:47:14.5944628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:47:14.5945036Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:14.5945177Z 2025-12-04T09:47:14.5945282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5945649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5945990Z res = mod(**inputs) 2025-12-04T09:47:14.5946347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5946740Z outputs = self.model.decoder( 2025-12-04T09:47:14.5947549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5947957Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5948317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5948697Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5949080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5949449Z return func(*args, **kwargs) 2025-12-04T09:47:14.5949858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.5950277Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.5950712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5951074Z return func(*args, **kwargs) 2025-12-04T09:47:14.5951456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:47:14.5951921Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:14.5952122Z 2025-12-04T09:47:14.5952232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5952581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5952928Z res = mod(**inputs) 2025-12-04T09:47:14.5953294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5953683Z outputs = self.model.decoder( 2025-12-04T09:47:14.5954071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5954467Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5954823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5955180Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5955563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5955939Z return func(*args, **kwargs) 2025-12-04T09:47:14.5956319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.5956740Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.5957160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5957529Z return func(*args, **kwargs) 2025-12-04T09:47:14.5957903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:47:14.5958305Z key_states = self.k_proj(current_states) 2025-12-04T09:47:14.5958450Z 2025-12-04T09:47:14.5958560Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5958926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5959244Z res = mod(**inputs) 2025-12-04T09:47:14.5959619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5960027Z outputs = self.model.decoder( 2025-12-04T09:47:14.5960467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5960879Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5961243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5961621Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5962021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5962411Z return func(*args, **kwargs) 2025-12-04T09:47:14.5962801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.5963231Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.5963634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5964043Z return func(*args, **kwargs) 2025-12-04T09:47:14.5964439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:47:14.5964853Z value_states = self.v_proj(current_states) 2025-12-04T09:47:14.5965029Z 2025-12-04T09:47:14.5965116Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.5965364Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5965732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5966053Z res = mod(**inputs) 2025-12-04T09:47:14.5966425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5966827Z outputs = self.model.decoder( 2025-12-04T09:47:14.5967213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5967627Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5967986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5968349Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5968732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5969113Z return func(*args, **kwargs) 2025-12-04T09:47:14.5969501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.5969922Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.5970318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5970698Z return func(*args, **kwargs) 2025-12-04T09:47:14.5971086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:47:14.5971507Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:14.5971963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:14.5972457Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:14.5972643Z 2025-12-04T09:47:14.5972757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5973126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5973445Z res = mod(**inputs) 2025-12-04T09:47:14.5973811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5974204Z outputs = self.model.decoder( 2025-12-04T09:47:14.5974589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5974969Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5975317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5975680Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5976064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5976433Z return func(*args, **kwargs) 2025-12-04T09:47:14.5976801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.5977198Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.5977593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5977984Z return func(*args, **kwargs) 2025-12-04T09:47:14.5978350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:47:14.5978748Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:14.5978888Z 2025-12-04T09:47:14.5979009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5979368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5979686Z res = mod(**inputs) 2025-12-04T09:47:14.5980051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5980441Z outputs = self.model.decoder( 2025-12-04T09:47:14.5980824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5981217Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5981577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5981942Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5982319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5982691Z return func(*args, **kwargs) 2025-12-04T09:47:14.5983072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.5983509Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.5983682Z 2025-12-04T09:47:14.5983788Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5984147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5984467Z res = mod(**inputs) 2025-12-04T09:47:14.5984830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5985264Z outputs = self.model.decoder( 2025-12-04T09:47:14.5985649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5986036Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5986379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5986741Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5987119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5987490Z return func(*args, **kwargs) 2025-12-04T09:47:14.5987862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.5988301Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.5988689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:14.5989027Z return self.act(input) 2025-12-04T09:47:14.5989147Z 2025-12-04T09:47:14.5989254Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5989665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5990043Z res = mod(**inputs) 2025-12-04T09:47:14.5990395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5990787Z outputs = self.model.decoder( 2025-12-04T09:47:14.5991168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5991558Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5991919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5992276Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5992670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5993038Z return func(*args, **kwargs) 2025-12-04T09:47:14.5993420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:47:14.5993823Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:14.5993960Z 2025-12-04T09:47:14.5994070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.5994414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.5994733Z res = mod(**inputs) 2025-12-04T09:47:14.5995115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.5995500Z outputs = self.model.decoder( 2025-12-04T09:47:14.5995881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.5996272Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.5996618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.5996968Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.5997344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5997714Z return func(*args, **kwargs) 2025-12-04T09:47:14.5998090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.5998496Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.5998893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.5999260Z return func(*args, **kwargs) 2025-12-04T09:47:14.5999629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:47:14.6000094Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:14.6000300Z 2025-12-04T09:47:14.6000472Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6000841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6001162Z res = mod(**inputs) 2025-12-04T09:47:14.6001559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6001999Z outputs = self.model.decoder( 2025-12-04T09:47:14.6002396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6002789Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6003147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6003515Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6003913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6004299Z return func(*args, **kwargs) 2025-12-04T09:47:14.6004672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6005082Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6005479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6005860Z return func(*args, **kwargs) 2025-12-04T09:47:14.6006221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:47:14.6006594Z key_states = self.k_proj(current_states) 2025-12-04T09:47:14.6006745Z 2025-12-04T09:47:14.6006847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6007186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6007496Z res = mod(**inputs) 2025-12-04T09:47:14.6007838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6008223Z outputs = self.model.decoder( 2025-12-04T09:47:14.6008599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6009003Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6009337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6009691Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6010061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6010421Z return func(*args, **kwargs) 2025-12-04T09:47:14.6010782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6011192Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6011577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6011929Z return func(*args, **kwargs) 2025-12-04T09:47:14.6012300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:47:14.6012696Z value_states = self.v_proj(current_states) 2025-12-04T09:47:14.6012833Z 2025-12-04T09:47:14.6012919Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.6013147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6013493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6013809Z res = mod(**inputs) 2025-12-04T09:47:14.6014157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6014540Z outputs = self.model.decoder( 2025-12-04T09:47:14.6014913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6015296Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6015629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6015985Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6016352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6016711Z return func(*args, **kwargs) 2025-12-04T09:47:14.6017095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6017507Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6017887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6018232Z return func(*args, **kwargs) 2025-12-04T09:47:14.6018590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:47:14.6018987Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:14.6019418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:14.6019880Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:14.6020062Z 2025-12-04T09:47:14.6020178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6020532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6020829Z res = mod(**inputs) 2025-12-04T09:47:14.6021185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6021565Z outputs = self.model.decoder( 2025-12-04T09:47:14.6021935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6022323Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6022664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6023013Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6023373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6023736Z return func(*args, **kwargs) 2025-12-04T09:47:14.6024105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6024508Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6024887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6025245Z return func(*args, **kwargs) 2025-12-04T09:47:14.6025610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:47:14.6026005Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:14.6026136Z 2025-12-04T09:47:14.6026240Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6026592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6026905Z res = mod(**inputs) 2025-12-04T09:47:14.6027256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6027641Z outputs = self.model.decoder( 2025-12-04T09:47:14.6028018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6028399Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6028729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6029080Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6029447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6029797Z return func(*args, **kwargs) 2025-12-04T09:47:14.6030166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.6030611Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.6030778Z 2025-12-04T09:47:14.6030891Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6031238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6031555Z res = mod(**inputs) 2025-12-04T09:47:14.6031917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6032306Z outputs = self.model.decoder( 2025-12-04T09:47:14.6032690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6033077Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6033434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6033777Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6034148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6034502Z return func(*args, **kwargs) 2025-12-04T09:47:14.6034868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.6035280Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.6035654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:14.6036003Z return self.act(input) 2025-12-04T09:47:14.6036111Z 2025-12-04T09:47:14.6036211Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6036562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6036874Z res = mod(**inputs) 2025-12-04T09:47:14.6037229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6037601Z outputs = self.model.decoder( 2025-12-04T09:47:14.6037982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6038371Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6038719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6039074Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6039453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6039822Z return func(*args, **kwargs) 2025-12-04T09:47:14.6040198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:47:14.6040687Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:14.6040863Z 2025-12-04T09:47:14.6040980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6041377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6041736Z res = mod(**inputs) 2025-12-04T09:47:14.6042135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6042560Z outputs = self.model.decoder( 2025-12-04T09:47:14.6042959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6043362Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6043706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6044077Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6044482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6044851Z return func(*args, **kwargs) 2025-12-04T09:47:14.6045234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6045655Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6046061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6046447Z return func(*args, **kwargs) 2025-12-04T09:47:14.6046832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:47:14.6047431Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:14.6047678Z 2025-12-04T09:47:14.6047795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6048148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6048471Z res = mod(**inputs) 2025-12-04T09:47:14.6048837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6049219Z outputs = self.model.decoder( 2025-12-04T09:47:14.6049602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6050022Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6050377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6050731Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6051111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6051483Z return func(*args, **kwargs) 2025-12-04T09:47:14.6051862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6052270Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6052664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6053034Z return func(*args, **kwargs) 2025-12-04T09:47:14.6053404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:47:14.6053807Z key_states = self.k_proj(current_states) 2025-12-04T09:47:14.6053951Z 2025-12-04T09:47:14.6054056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6054417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6054730Z res = mod(**inputs) 2025-12-04T09:47:14.6055095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6055487Z outputs = self.model.decoder( 2025-12-04T09:47:14.6055874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6056244Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6056580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6056936Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6057299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6057663Z return func(*args, **kwargs) 2025-12-04T09:47:14.6058036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6058483Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6058865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6059229Z return func(*args, **kwargs) 2025-12-04T09:47:14.6059586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:47:14.6059963Z value_states = self.v_proj(current_states) 2025-12-04T09:47:14.6060107Z 2025-12-04T09:47:14.6060216Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.6060445Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6060786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6061086Z res = mod(**inputs) 2025-12-04T09:47:14.6061451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6061833Z outputs = self.model.decoder( 2025-12-04T09:47:14.6062196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6062570Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6062901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6063251Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6063622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6063975Z return func(*args, **kwargs) 2025-12-04T09:47:14.6064333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6064728Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6065098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6065448Z return func(*args, **kwargs) 2025-12-04T09:47:14.6065805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:47:14.6066196Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:14.6066629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:14.6067099Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:14.6067275Z 2025-12-04T09:47:14.6067382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6067729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6068040Z res = mod(**inputs) 2025-12-04T09:47:14.6068397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6068781Z outputs = self.model.decoder( 2025-12-04T09:47:14.6069147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6069531Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6069869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6070214Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6070587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6070955Z return func(*args, **kwargs) 2025-12-04T09:47:14.6071370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6071776Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6072157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6072518Z return func(*args, **kwargs) 2025-12-04T09:47:14.6072878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:47:14.6073268Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:14.6073410Z 2025-12-04T09:47:14.6073513Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6073877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6074179Z res = mod(**inputs) 2025-12-04T09:47:14.6074556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6074999Z outputs = self.model.decoder( 2025-12-04T09:47:14.6075381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6075762Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6076111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6076475Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6076850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6077240Z return func(*args, **kwargs) 2025-12-04T09:47:14.6077620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.6078057Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.6078237Z 2025-12-04T09:47:14.6078346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6078716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6079055Z res = mod(**inputs) 2025-12-04T09:47:14.6079428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6079831Z outputs = self.model.decoder( 2025-12-04T09:47:14.6080231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6080707Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6081080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6081457Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6081849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6082234Z return func(*args, **kwargs) 2025-12-04T09:47:14.6082620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.6083051Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.6083434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:14.6083766Z return self.act(input) 2025-12-04T09:47:14.6083883Z 2025-12-04T09:47:14.6083988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6084340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6084655Z res = mod(**inputs) 2025-12-04T09:47:14.6085001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6085390Z outputs = self.model.decoder( 2025-12-04T09:47:14.6085790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6086165Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6086508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6086859Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6087225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6087598Z return func(*args, **kwargs) 2025-12-04T09:47:14.6087965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:47:14.6088365Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:14.6088497Z 2025-12-04T09:47:14.6088619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6088960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6089276Z res = mod(**inputs) 2025-12-04T09:47:14.6089628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6090006Z outputs = self.model.decoder( 2025-12-04T09:47:14.6090380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6090781Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6091124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6091468Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6091839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6092206Z return func(*args, **kwargs) 2025-12-04T09:47:14.6092572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6092983Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6093372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6093733Z return func(*args, **kwargs) 2025-12-04T09:47:14.6094093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:47:14.6094552Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:14.6094746Z 2025-12-04T09:47:14.6094854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6095205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6095520Z res = mod(**inputs) 2025-12-04T09:47:14.6095889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6096282Z outputs = self.model.decoder( 2025-12-04T09:47:14.6096661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6097053Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6097403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6097775Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6098193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6098573Z return func(*args, **kwargs) 2025-12-04T09:47:14.6098958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6099391Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6099789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6100163Z return func(*args, **kwargs) 2025-12-04T09:47:14.6100541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:47:14.6100931Z key_states = self.k_proj(current_states) 2025-12-04T09:47:14.6101074Z 2025-12-04T09:47:14.6101177Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6101551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6101873Z res = mod(**inputs) 2025-12-04T09:47:14.6102247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6102641Z outputs = self.model.decoder( 2025-12-04T09:47:14.6103029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6103412Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6103762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6104125Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6104497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6104918Z return func(*args, **kwargs) 2025-12-04T09:47:14.6105294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6105710Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6106107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6106472Z return func(*args, **kwargs) 2025-12-04T09:47:14.6106849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:47:14.6107253Z value_states = self.v_proj(current_states) 2025-12-04T09:47:14.6107397Z 2025-12-04T09:47:14.6107480Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.6107730Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6108081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6108404Z res = mod(**inputs) 2025-12-04T09:47:14.6108741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6109102Z outputs = self.model.decoder( 2025-12-04T09:47:14.6109466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6109833Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6110164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6110508Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6110868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6111213Z return func(*args, **kwargs) 2025-12-04T09:47:14.6111571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6111966Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6112337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6112688Z return func(*args, **kwargs) 2025-12-04T09:47:14.6113061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:47:14.6113456Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:14.6113870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:14.6114332Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:14.6114511Z 2025-12-04T09:47:14.6114612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6114974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6115274Z res = mod(**inputs) 2025-12-04T09:47:14.6115637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6116017Z outputs = self.model.decoder( 2025-12-04T09:47:14.6116385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6116771Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6117120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6117483Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6117854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6118232Z return func(*args, **kwargs) 2025-12-04T09:47:14.6118600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6119003Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6119379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6119742Z return func(*args, **kwargs) 2025-12-04T09:47:14.6120117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:47:14.6120620Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:14.6120787Z 2025-12-04T09:47:14.6120903Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6121312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6121685Z res = mod(**inputs) 2025-12-04T09:47:14.6122049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6122448Z outputs = self.model.decoder( 2025-12-04T09:47:14.6122842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6123234Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6123592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6123971Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6124356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6124725Z return func(*args, **kwargs) 2025-12-04T09:47:14.6125108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.6125553Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.6125724Z 2025-12-04T09:47:14.6125835Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6126191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6126513Z res = mod(**inputs) 2025-12-04T09:47:14.6126908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6127296Z outputs = self.model.decoder( 2025-12-04T09:47:14.6127687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6128084Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6128439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6128810Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6129184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6129564Z return func(*args, **kwargs) 2025-12-04T09:47:14.6129948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.6130389Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.6130776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:14.6131123Z return self.act(input) 2025-12-04T09:47:14.6131235Z 2025-12-04T09:47:14.6131339Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6131699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6132042Z res = mod(**inputs) 2025-12-04T09:47:14.6132408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6132803Z outputs = self.model.decoder( 2025-12-04T09:47:14.6133192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6133581Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6133923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6134285Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6134665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6135036Z return func(*args, **kwargs) 2025-12-04T09:47:14.6135404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:47:14.6135804Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:14.6135942Z 2025-12-04T09:47:14.6136053Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6136401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6136722Z res = mod(**inputs) 2025-12-04T09:47:14.6137092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6137488Z outputs = self.model.decoder( 2025-12-04T09:47:14.6137865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6138268Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6138606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6138962Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6139324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6139685Z return func(*args, **kwargs) 2025-12-04T09:47:14.6140055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6140459Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6140865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6141231Z return func(*args, **kwargs) 2025-12-04T09:47:14.6141606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:47:14.6142061Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:14.6142270Z 2025-12-04T09:47:14.6142373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6142738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6143047Z res = mod(**inputs) 2025-12-04T09:47:14.6143411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6143801Z outputs = self.model.decoder( 2025-12-04T09:47:14.6144180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6144559Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6144900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6145305Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6145672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6146044Z return func(*args, **kwargs) 2025-12-04T09:47:14.6146411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6146818Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6147346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6147721Z return func(*args, **kwargs) 2025-12-04T09:47:14.6148094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:47:14.6148491Z key_states = self.k_proj(current_states) 2025-12-04T09:47:14.6148626Z 2025-12-04T09:47:14.6148730Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6149088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6149414Z res = mod(**inputs) 2025-12-04T09:47:14.6149783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6150176Z outputs = self.model.decoder( 2025-12-04T09:47:14.6150556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6150944Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6151276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6151627Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6151994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6152353Z return func(*args, **kwargs) 2025-12-04T09:47:14.6152713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6153120Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6153505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6153860Z return func(*args, **kwargs) 2025-12-04T09:47:14.6154262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:47:14.6154656Z value_states = self.v_proj(current_states) 2025-12-04T09:47:14.6154792Z 2025-12-04T09:47:14.6154877Z cudagraph partition due to non gpu ops 2025-12-04T09:47:14.6155103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6155445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6155759Z res = mod(**inputs) 2025-12-04T09:47:14.6156115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6156518Z outputs = self.model.decoder( 2025-12-04T09:47:14.6156895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6157316Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6157651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6158007Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6158380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6158743Z return func(*args, **kwargs) 2025-12-04T09:47:14.6159107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6159533Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6159921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6160273Z return func(*args, **kwargs) 2025-12-04T09:47:14.6160697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:47:14.6161104Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:14.6161572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:14.6162092Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:14.6162288Z 2025-12-04T09:47:14.6162395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6162775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6163099Z res = mod(**inputs) 2025-12-04T09:47:14.6163468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6163852Z outputs = self.model.decoder( 2025-12-04T09:47:14.6164268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6164687Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6165073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6165469Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6165883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6166286Z return func(*args, **kwargs) 2025-12-04T09:47:14.6166697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:47:14.6167153Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:14.6167583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6167987Z return func(*args, **kwargs) 2025-12-04T09:47:14.6168395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:47:14.6168853Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:14.6169003Z 2025-12-04T09:47:14.6169116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6169501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6169847Z res = mod(**inputs) 2025-12-04T09:47:14.6170241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6170673Z outputs = self.model.decoder( 2025-12-04T09:47:14.6171050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6171440Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6171820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6172178Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6172544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6172907Z return func(*args, **kwargs) 2025-12-04T09:47:14.6173265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.6173686Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.6173869Z 2025-12-04T09:47:14.6173977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6174325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6174627Z res = mod(**inputs) 2025-12-04T09:47:14.6174983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6175422Z outputs = self.model.decoder( 2025-12-04T09:47:14.6175790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6176177Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6176519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6176869Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6177232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6177596Z return func(*args, **kwargs) 2025-12-04T09:47:14.6177968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:47:14.6178385Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:14.6178766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:14.6179105Z return self.act(input) 2025-12-04T09:47:14.6179212Z 2025-12-04T09:47:14.6179324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6179664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6179978Z res = mod(**inputs) 2025-12-04T09:47:14.6180336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:47:14.6180723Z outputs = self.model.decoder( 2025-12-04T09:47:14.6181089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:47:14.6181467Z layer_outputs = decoder_layer( 2025-12-04T09:47:14.6181809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:14.6182154Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:14.6182537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:14.6182900Z return func(*args, **kwargs) 2025-12-04T09:47:14.6183268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:47:14.6183650Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:14.6183790Z 2025-12-04T09:47:14.6183893Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6184264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6184569Z res = mod(**inputs) 2025-12-04T09:47:14.6184922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1696, in forward 2025-12-04T09:47:14.6185323Z logits = self.lm_head(outputs[0]) 2025-12-04T09:47:14.6185452Z 2025-12-04T09:47:14.6185564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:14.6185904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:14.6186211Z res = mod(**inputs) 2025-12-04T09:47:14.6186565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1702, in forward 2025-12-04T09:47:14.6187011Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:47:14.6187222Z 2025-12-04T09:47:23.8707501Z Compilation time (from dynamo_timed): 12.873412884 2025-12-04T09:47:23.9017351Z pass 2025-12-04T09:47:23.9022024Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:23.9022970Z TIMING: _recursive_pre_grad_passes:0.00526 _recursive_joint_graph_passes:0.24272 _recursive_post_grad_passes:0.04433 async_compile.wait:0.73582 code_gen:9.01094 inductor_compile:9.70339 backend_compile:11.42847 gc:0.00105 entire_frame_compile:12.87341 total_wall_time:12.87341 2025-12-04T09:47:23.9024071Z STATS: call_* op count: 180 | FakeTensorMode.__torch_dispatch__:4080 | FakeTensor.__torch_dispatch__:2342 | ProxyTorchDispatchMode.__torch_dispatch__:1092 2025-12-04T09:47:23.9024563Z Dynamo produced 1 graphs covering 180 ops with 0 graph breaks (0 unique) 2025-12-04T09:47:26.0741910Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:47:26.0742876Z import pynvml # type: ignore[import] 2025-12-04T09:47:29.4360626Z 2025-12-04T09:47:32.5099392Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:47:32.5099885Z loading model: 0it [00:03, ?it/s] 2025-12-04T09:47:32.5119397Z cpu eval PegasusForCausalLM 2025-12-04T09:47:32.8715109Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:33.0205770Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:33.1599932Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:40.7627470Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7627817Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7628071Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7628341Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7628591Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7628831Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7629072Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7629306Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7629566Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7629807Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7630399Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7630685Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7630962Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7631420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7631800Z res = mod(**inputs) 2025-12-04T09:47:40.7632265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7632817Z outputs = self.model.decoder( 2025-12-04T09:47:40.7633274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7633754Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7634924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7635394Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7635827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7636247Z return func(*args, **kwargs) 2025-12-04T09:47:40.7636704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7637187Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7637708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7638138Z return func(*args, **kwargs) 2025-12-04T09:47:40.7638572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.7639337Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.7639595Z 2025-12-04T09:47:40.7639730Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7640137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7640805Z res = mod(**inputs) 2025-12-04T09:47:40.7641240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7641736Z outputs = self.model.decoder( 2025-12-04T09:47:40.7642181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7642615Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7642995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7643396Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7643817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7644231Z return func(*args, **kwargs) 2025-12-04T09:47:40.7644644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7645099Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7645540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7645951Z return func(*args, **kwargs) 2025-12-04T09:47:40.7646361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.7646818Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.7646965Z 2025-12-04T09:47:40.7647222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7647628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7648042Z res = mod(**inputs) 2025-12-04T09:47:40.7648458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7648894Z outputs = self.model.decoder( 2025-12-04T09:47:40.7649310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7649743Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7650124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7650563Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7650980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7651448Z return func(*args, **kwargs) 2025-12-04T09:47:40.7651868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7652318Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7652751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7653200Z return func(*args, **kwargs) 2025-12-04T09:47:40.7653621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.7654092Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.7654253Z 2025-12-04T09:47:40.7654344Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7654605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7654997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7655337Z res = mod(**inputs) 2025-12-04T09:47:40.7655723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7656128Z outputs = self.model.decoder( 2025-12-04T09:47:40.7656525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7656936Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7657314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7657710Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7658109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7658522Z return func(*args, **kwargs) 2025-12-04T09:47:40.7658939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7659394Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7659801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7660248Z return func(*args, **kwargs) 2025-12-04T09:47:40.7660662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.7661114Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.7661602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.7662131Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.7662329Z 2025-12-04T09:47:40.7662454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7662845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7663223Z res = mod(**inputs) 2025-12-04T09:47:40.7663629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7664073Z outputs = self.model.decoder( 2025-12-04T09:47:40.7664507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7665219Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7665602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7666015Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7666424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7666852Z return func(*args, **kwargs) 2025-12-04T09:47:40.7667271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7667712Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7668168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7668570Z return func(*args, **kwargs) 2025-12-04T09:47:40.7668984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.7669441Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.7669598Z 2025-12-04T09:47:40.7669712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7670107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7670466Z res = mod(**inputs) 2025-12-04T09:47:40.7670879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7671320Z outputs = self.model.decoder( 2025-12-04T09:47:40.7671753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7672201Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7672581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7672984Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7673403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7673819Z return func(*args, **kwargs) 2025-12-04T09:47:40.7674248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7674741Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7674934Z 2025-12-04T09:47:40.7675051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7675451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7675805Z res = mod(**inputs) 2025-12-04T09:47:40.7676216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7676654Z outputs = self.model.decoder( 2025-12-04T09:47:40.7677091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7677538Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7677920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7678329Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7678777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7679194Z return func(*args, **kwargs) 2025-12-04T09:47:40.7679619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7680118Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7680637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.7681036Z return self.act(input) 2025-12-04T09:47:40.7681219Z 2025-12-04T09:47:40.7681336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7681742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7682105Z res = mod(**inputs) 2025-12-04T09:47:40.7682542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7682994Z outputs = self.model.decoder( 2025-12-04T09:47:40.7683433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7683873Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7684256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7684663Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7685097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7685471Z return func(*args, **kwargs) 2025-12-04T09:47:40.7685866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.7686265Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.7686399Z 2025-12-04T09:47:40.7686509Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7686859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7687173Z res = mod(**inputs) 2025-12-04T09:47:40.7687548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7687945Z outputs = self.model.decoder( 2025-12-04T09:47:40.7688343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7688739Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7689084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7689444Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7689833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7690233Z return func(*args, **kwargs) 2025-12-04T09:47:40.7690650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7691092Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7691513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7691897Z return func(*args, **kwargs) 2025-12-04T09:47:40.7692283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.7692764Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.7692978Z 2025-12-04T09:47:40.7693094Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7693464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7693778Z res = mod(**inputs) 2025-12-04T09:47:40.7694156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7694558Z outputs = self.model.decoder( 2025-12-04T09:47:40.7694955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7695352Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7695720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7696080Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7696462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7696835Z return func(*args, **kwargs) 2025-12-04T09:47:40.7697220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7697644Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7698041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7698422Z return func(*args, **kwargs) 2025-12-04T09:47:40.7698816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.7699248Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.7699395Z 2025-12-04T09:47:40.7699503Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7699889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7700236Z res = mod(**inputs) 2025-12-04T09:47:40.7700633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7701072Z outputs = self.model.decoder( 2025-12-04T09:47:40.7701473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7701877Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7702235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7702627Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7703037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7703431Z return func(*args, **kwargs) 2025-12-04T09:47:40.7703850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7704302Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7704733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7705133Z return func(*args, **kwargs) 2025-12-04T09:47:40.7705547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.7705987Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.7706143Z 2025-12-04T09:47:40.7706241Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7706495Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7706997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7707352Z res = mod(**inputs) 2025-12-04T09:47:40.7707750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7708208Z outputs = self.model.decoder( 2025-12-04T09:47:40.7708641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7709080Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7709456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7709857Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7710275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7710733Z return func(*args, **kwargs) 2025-12-04T09:47:40.7711148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7711620Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7712058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7712459Z return func(*args, **kwargs) 2025-12-04T09:47:40.7712870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.7713321Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.7713793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.7714337Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.7714543Z 2025-12-04T09:47:40.7714652Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7715037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7715378Z res = mod(**inputs) 2025-12-04T09:47:40.7715797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7716242Z outputs = self.model.decoder( 2025-12-04T09:47:40.7716676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7717097Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7717476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7717872Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7718282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7718695Z return func(*args, **kwargs) 2025-12-04T09:47:40.7719112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7719567Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7719987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7720396Z return func(*args, **kwargs) 2025-12-04T09:47:40.7720894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.7721348Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.7721503Z 2025-12-04T09:47:40.7721624Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7722027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7722383Z res = mod(**inputs) 2025-12-04T09:47:40.7722760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7723174Z outputs = self.model.decoder( 2025-12-04T09:47:40.7723614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7724048Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7724421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7724816Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7725206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7725598Z return func(*args, **kwargs) 2025-12-04T09:47:40.7726007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7726482Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7726665Z 2025-12-04T09:47:40.7726802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7727188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7727537Z res = mod(**inputs) 2025-12-04T09:47:40.7727936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7728368Z outputs = self.model.decoder( 2025-12-04T09:47:40.7728785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7729237Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7729614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7730000Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7730414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7730814Z return func(*args, **kwargs) 2025-12-04T09:47:40.7731231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7731694Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7732114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.7732487Z return self.act(input) 2025-12-04T09:47:40.7732605Z 2025-12-04T09:47:40.7732727Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7733108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7733454Z res = mod(**inputs) 2025-12-04T09:47:40.7733856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7734280Z outputs = self.model.decoder( 2025-12-04T09:47:40.7734704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7735133Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7735509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7735903Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7736321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7736740Z return func(*args, **kwargs) 2025-12-04T09:47:40.7737150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.7737590Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.7737745Z 2025-12-04T09:47:40.7737860Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7738283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7738627Z res = mod(**inputs) 2025-12-04T09:47:40.7739030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7739467Z outputs = self.model.decoder( 2025-12-04T09:47:40.7739883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7740316Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7740717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7741113Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7741533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7741945Z return func(*args, **kwargs) 2025-12-04T09:47:40.7742375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7742826Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7743246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7743653Z return func(*args, **kwargs) 2025-12-04T09:47:40.7744068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.7744596Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.7744823Z 2025-12-04T09:47:40.7744936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7745326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7745680Z res = mod(**inputs) 2025-12-04T09:47:40.7746054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7746464Z outputs = self.model.decoder( 2025-12-04T09:47:40.7746866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7747460Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7747821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7748209Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7748601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7749021Z return func(*args, **kwargs) 2025-12-04T09:47:40.7749446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7749910Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7750348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7750757Z return func(*args, **kwargs) 2025-12-04T09:47:40.7751155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.7751572Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.7751713Z 2025-12-04T09:47:40.7751832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7752190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7752518Z res = mod(**inputs) 2025-12-04T09:47:40.7752905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7753313Z outputs = self.model.decoder( 2025-12-04T09:47:40.7753780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7754191Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7754545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7754904Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7755291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7755706Z return func(*args, **kwargs) 2025-12-04T09:47:40.7756090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7756550Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7756960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7757357Z return func(*args, **kwargs) 2025-12-04T09:47:40.7757749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.7758202Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.7758353Z 2025-12-04T09:47:40.7758465Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7758723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7759161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7759514Z res = mod(**inputs) 2025-12-04T09:47:40.7759914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7760339Z outputs = self.model.decoder( 2025-12-04T09:47:40.7760832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7761275Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7761665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7762053Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7762480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7769712Z return func(*args, **kwargs) 2025-12-04T09:47:40.7770308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7770860Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7771323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7771749Z return func(*args, **kwargs) 2025-12-04T09:47:40.7772182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.7772646Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.7773139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.7773673Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.7773882Z 2025-12-04T09:47:40.7774009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7774417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7774777Z res = mod(**inputs) 2025-12-04T09:47:40.7775197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7775651Z outputs = self.model.decoder( 2025-12-04T09:47:40.7776195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7776647Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7777040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7777445Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7777877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7778344Z return func(*args, **kwargs) 2025-12-04T09:47:40.7778773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7779267Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7779715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7780118Z return func(*args, **kwargs) 2025-12-04T09:47:40.7780530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.7780973Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.7781124Z 2025-12-04T09:47:40.7781250Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7781637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7782033Z res = mod(**inputs) 2025-12-04T09:47:40.7782447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7782888Z outputs = self.model.decoder( 2025-12-04T09:47:40.7783314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7783753Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7784139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7784538Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7784948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7785354Z return func(*args, **kwargs) 2025-12-04T09:47:40.7785773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7786227Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7786415Z 2025-12-04T09:47:40.7786526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7786899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7787232Z res = mod(**inputs) 2025-12-04T09:47:40.7787606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7788019Z outputs = self.model.decoder( 2025-12-04T09:47:40.7788423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7788848Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7789232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7789635Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7790046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7790428Z return func(*args, **kwargs) 2025-12-04T09:47:40.7790875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7791359Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7791785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.7792155Z return self.act(input) 2025-12-04T09:47:40.7792285Z 2025-12-04T09:47:40.7792398Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7792791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7793152Z res = mod(**inputs) 2025-12-04T09:47:40.7793562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7793975Z outputs = self.model.decoder( 2025-12-04T09:47:40.7794400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7794806Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7795165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7795539Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7795927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7796347Z return func(*args, **kwargs) 2025-12-04T09:47:40.7796767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.7797252Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.7797399Z 2025-12-04T09:47:40.7797512Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7797901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7798248Z res = mod(**inputs) 2025-12-04T09:47:40.7798657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7799084Z outputs = self.model.decoder( 2025-12-04T09:47:40.7799507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7799937Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7800315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7800954Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7801392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7801818Z return func(*args, **kwargs) 2025-12-04T09:47:40.7802238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T09:47:40.7802665Z hidden_states = residual + hidden_states 2025-12-04T09:47:40.7802818Z 2025-12-04T09:47:40.7802943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7803333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7803691Z res = mod(**inputs) 2025-12-04T09:47:40.7804171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7804623Z outputs = self.model.decoder( 2025-12-04T09:47:40.7805056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7805575Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7805967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7806370Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7806819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7807239Z return func(*args, **kwargs) 2025-12-04T09:47:40.7807667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7808135Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7808587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7809029Z return func(*args, **kwargs) 2025-12-04T09:47:40.7809456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.7809999Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.7810239Z 2025-12-04T09:47:40.7810352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7810731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7811084Z res = mod(**inputs) 2025-12-04T09:47:40.7811504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7811950Z outputs = self.model.decoder( 2025-12-04T09:47:40.7812393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7812847Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7813229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7813630Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7814018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7814394Z return func(*args, **kwargs) 2025-12-04T09:47:40.7814793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7815226Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7815626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7816010Z return func(*args, **kwargs) 2025-12-04T09:47:40.7816407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.7816826Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.7816964Z 2025-12-04T09:47:40.7817077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7817468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7817818Z res = mod(**inputs) 2025-12-04T09:47:40.7818226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7818651Z outputs = self.model.decoder( 2025-12-04T09:47:40.7819075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7819508Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7819885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7820283Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7820721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7821125Z return func(*args, **kwargs) 2025-12-04T09:47:40.7821565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7822026Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7822449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7822852Z return func(*args, **kwargs) 2025-12-04T09:47:40.7823268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.7823714Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.7823891Z 2025-12-04T09:47:40.7823982Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7824252Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7824637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7824994Z res = mod(**inputs) 2025-12-04T09:47:40.7825400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7825830Z outputs = self.model.decoder( 2025-12-04T09:47:40.7826257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7826679Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7827055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7827465Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7827875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7828281Z return func(*args, **kwargs) 2025-12-04T09:47:40.7828698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7829160Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7829593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7830016Z return func(*args, **kwargs) 2025-12-04T09:47:40.7830445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.7830898Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.7831387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.7831926Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.7832130Z 2025-12-04T09:47:40.7832262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7832648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7832989Z res = mod(**inputs) 2025-12-04T09:47:40.7833394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7833835Z outputs = self.model.decoder( 2025-12-04T09:47:40.7834268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7834708Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7835084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7835481Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7835883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7836285Z return func(*args, **kwargs) 2025-12-04T09:47:40.7836725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7837175Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7837610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7838016Z return func(*args, **kwargs) 2025-12-04T09:47:40.7838434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.7838886Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.7839067Z 2025-12-04T09:47:40.7839183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7839583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7839944Z res = mod(**inputs) 2025-12-04T09:47:40.7840367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7840917Z outputs = self.model.decoder( 2025-12-04T09:47:40.7841364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7841804Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7842256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7842675Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7843124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7843523Z return func(*args, **kwargs) 2025-12-04T09:47:40.7843954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7844453Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7844644Z 2025-12-04T09:47:40.7844771Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7845162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7845526Z res = mod(**inputs) 2025-12-04T09:47:40.7845944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7846387Z outputs = self.model.decoder( 2025-12-04T09:47:40.7846832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7847473Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7847869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7848274Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7848702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7849126Z return func(*args, **kwargs) 2025-12-04T09:47:40.7849549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7850050Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7850486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.7850872Z return self.act(input) 2025-12-04T09:47:40.7851006Z 2025-12-04T09:47:40.7851114Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7851479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7851812Z res = mod(**inputs) 2025-12-04T09:47:40.7852189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7852710Z outputs = self.model.decoder( 2025-12-04T09:47:40.7853108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7853506Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7853853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7854224Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7854613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7855035Z return func(*args, **kwargs) 2025-12-04T09:47:40.7855424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.7855886Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.7856026Z 2025-12-04T09:47:40.7856145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7856505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7856834Z res = mod(**inputs) 2025-12-04T09:47:40.7857212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7857623Z outputs = self.model.decoder( 2025-12-04T09:47:40.7858018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7858451Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7858807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7859171Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7859549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7859921Z return func(*args, **kwargs) 2025-12-04T09:47:40.7860303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7860717Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7861115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7861494Z return func(*args, **kwargs) 2025-12-04T09:47:40.7861890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.7862367Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.7862586Z 2025-12-04T09:47:40.7862700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7863092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7863430Z res = mod(**inputs) 2025-12-04T09:47:40.7863813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7864223Z outputs = self.model.decoder( 2025-12-04T09:47:40.7864627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7865018Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7865370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7865734Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7866119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7866497Z return func(*args, **kwargs) 2025-12-04T09:47:40.7866908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7867338Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7867740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7868118Z return func(*args, **kwargs) 2025-12-04T09:47:40.7868510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.7868942Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.7869081Z 2025-12-04T09:47:40.7869187Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7869553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7869899Z res = mod(**inputs) 2025-12-04T09:47:40.7870273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7870683Z outputs = self.model.decoder( 2025-12-04T09:47:40.7871082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7871485Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7871833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7872220Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7872611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7872992Z return func(*args, **kwargs) 2025-12-04T09:47:40.7873379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7873805Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7874212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7874582Z return func(*args, **kwargs) 2025-12-04T09:47:40.7874972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.7875394Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.7875536Z 2025-12-04T09:47:40.7875629Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7875869Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7876233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7876558Z res = mod(**inputs) 2025-12-04T09:47:40.7876933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7877342Z outputs = self.model.decoder( 2025-12-04T09:47:40.7877741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7878146Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7878495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7878864Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7879247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7879633Z return func(*args, **kwargs) 2025-12-04T09:47:40.7880020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7880522Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7880992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7881411Z return func(*args, **kwargs) 2025-12-04T09:47:40.7881838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.7882303Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.7882761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.7883248Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.7883468Z 2025-12-04T09:47:40.7883574Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7883943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7884288Z res = mod(**inputs) 2025-12-04T09:47:40.7884665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7885079Z outputs = self.model.decoder( 2025-12-04T09:47:40.7885480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7885869Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7886218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7886601Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7886982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7887341Z return func(*args, **kwargs) 2025-12-04T09:47:40.7887736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7888144Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7888522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7888890Z return func(*args, **kwargs) 2025-12-04T09:47:40.7889272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.7889683Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.7889822Z 2025-12-04T09:47:40.7889930Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7890300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7890627Z res = mod(**inputs) 2025-12-04T09:47:40.7891012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7891411Z outputs = self.model.decoder( 2025-12-04T09:47:40.7891797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7892184Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7892516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7892885Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7893264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7893635Z return func(*args, **kwargs) 2025-12-04T09:47:40.7894013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7894453Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7894624Z 2025-12-04T09:47:40.7894736Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7895127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7895440Z res = mod(**inputs) 2025-12-04T09:47:40.7895801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7896191Z outputs = self.model.decoder( 2025-12-04T09:47:40.7896564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7896955Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7897314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7897678Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7898067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7898442Z return func(*args, **kwargs) 2025-12-04T09:47:40.7898833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7899260Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7899642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.7899985Z return self.act(input) 2025-12-04T09:47:40.7900099Z 2025-12-04T09:47:40.7900212Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7900592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7900917Z res = mod(**inputs) 2025-12-04T09:47:40.7901286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7901676Z outputs = self.model.decoder( 2025-12-04T09:47:40.7902069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7902469Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7902819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7903177Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7903554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7903933Z return func(*args, **kwargs) 2025-12-04T09:47:40.7904320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.7904795Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.7904938Z 2025-12-04T09:47:40.7905043Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7905399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7905709Z res = mod(**inputs) 2025-12-04T09:47:40.7906080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7906477Z outputs = self.model.decoder( 2025-12-04T09:47:40.7906866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7907258Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7907609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7907979Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7908350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7908724Z return func(*args, **kwargs) 2025-12-04T09:47:40.7909139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T09:47:40.7909559Z hidden_states = residual + hidden_states 2025-12-04T09:47:40.7909695Z 2025-12-04T09:47:40.7909802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7910171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7910521Z res = mod(**inputs) 2025-12-04T09:47:40.7910917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7911379Z outputs = self.model.decoder( 2025-12-04T09:47:40.7911780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7912207Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7912562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7912940Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7913333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7913720Z return func(*args, **kwargs) 2025-12-04T09:47:40.7914116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7914573Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7914986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7915363Z return func(*args, **kwargs) 2025-12-04T09:47:40.7915748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.7916232Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.7916441Z 2025-12-04T09:47:40.7916555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7916914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7917240Z res = mod(**inputs) 2025-12-04T09:47:40.7917620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7918028Z outputs = self.model.decoder( 2025-12-04T09:47:40.7918420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7918831Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7919196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7919560Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7919961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7920361Z return func(*args, **kwargs) 2025-12-04T09:47:40.7920862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7921328Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7921786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7922224Z return func(*args, **kwargs) 2025-12-04T09:47:40.7922665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.7923112Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.7923263Z 2025-12-04T09:47:40.7923371Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7923779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7924107Z res = mod(**inputs) 2025-12-04T09:47:40.7924513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7924952Z outputs = self.model.decoder( 2025-12-04T09:47:40.7925382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7925834Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7926214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7926612Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7927040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7927423Z return func(*args, **kwargs) 2025-12-04T09:47:40.7927846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7928301Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7928730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7929132Z return func(*args, **kwargs) 2025-12-04T09:47:40.7929550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.7930015Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.7930168Z 2025-12-04T09:47:40.7930258Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7930521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7930908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7931251Z res = mod(**inputs) 2025-12-04T09:47:40.7931667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7932115Z outputs = self.model.decoder( 2025-12-04T09:47:40.7932539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7932964Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7933343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7933763Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7934176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7934591Z return func(*args, **kwargs) 2025-12-04T09:47:40.7935011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7935478Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7935919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7936322Z return func(*args, **kwargs) 2025-12-04T09:47:40.7936742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.7937199Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.7937676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.7938196Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.7938393Z 2025-12-04T09:47:40.7938514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7938927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7939281Z res = mod(**inputs) 2025-12-04T09:47:40.7939698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7940145Z outputs = self.model.decoder( 2025-12-04T09:47:40.7940583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7941017Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7941421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7941809Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7942254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7942660Z return func(*args, **kwargs) 2025-12-04T09:47:40.7943083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7943532Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7943967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7944369Z return func(*args, **kwargs) 2025-12-04T09:47:40.7944786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.7945248Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.7945404Z 2025-12-04T09:47:40.7945519Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7945920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7946265Z res = mod(**inputs) 2025-12-04T09:47:40.7946683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7947324Z outputs = self.model.decoder( 2025-12-04T09:47:40.7947760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7948190Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7948575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7948981Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7949386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7949797Z return func(*args, **kwargs) 2025-12-04T09:47:40.7950218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7950705Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7950897Z 2025-12-04T09:47:40.7951016Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7951404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7951755Z res = mod(**inputs) 2025-12-04T09:47:40.7952160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7952593Z outputs = self.model.decoder( 2025-12-04T09:47:40.7953038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7953473Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7953850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7954252Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7954747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7955149Z return func(*args, **kwargs) 2025-12-04T09:47:40.7955552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.7956030Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.7956450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.7956853Z return self.act(input) 2025-12-04T09:47:40.7956970Z 2025-12-04T09:47:40.7957078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7957480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7957828Z res = mod(**inputs) 2025-12-04T09:47:40.7958222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7958655Z outputs = self.model.decoder( 2025-12-04T09:47:40.7959078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7959513Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7959892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7960335Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7960835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7961270Z return func(*args, **kwargs) 2025-12-04T09:47:40.7961703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.7962159Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.7962304Z 2025-12-04T09:47:40.7962421Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7962789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7963130Z res = mod(**inputs) 2025-12-04T09:47:40.7963525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7963951Z outputs = self.model.decoder( 2025-12-04T09:47:40.7964360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7964781Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7965149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7965524Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7965934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7966316Z return func(*args, **kwargs) 2025-12-04T09:47:40.7966710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7967136Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7967549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7967936Z return func(*args, **kwargs) 2025-12-04T09:47:40.7968320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.7968805Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.7969023Z 2025-12-04T09:47:40.7969131Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7969558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7969901Z res = mod(**inputs) 2025-12-04T09:47:40.7970312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7970723Z outputs = self.model.decoder( 2025-12-04T09:47:40.7971128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7971552Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7971917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7972295Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7972696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7973087Z return func(*args, **kwargs) 2025-12-04T09:47:40.7973483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7973914Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7974314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7974697Z return func(*args, **kwargs) 2025-12-04T09:47:40.7975113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.7975527Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.7975665Z 2025-12-04T09:47:40.7975771Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7976136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7976462Z res = mod(**inputs) 2025-12-04T09:47:40.7976831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7977239Z outputs = self.model.decoder( 2025-12-04T09:47:40.7977638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7978046Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7978395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7978772Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7979157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7979536Z return func(*args, **kwargs) 2025-12-04T09:47:40.7979921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7980339Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7980737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7981108Z return func(*args, **kwargs) 2025-12-04T09:47:40.7981503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.7981927Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.7982150Z 2025-12-04T09:47:40.7982254Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.7982508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7982869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7983196Z res = mod(**inputs) 2025-12-04T09:47:40.7983593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7984018Z outputs = self.model.decoder( 2025-12-04T09:47:40.7984485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7984916Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7985272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7985653Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7986065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7986444Z return func(*args, **kwargs) 2025-12-04T09:47:40.7986849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7987268Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7987668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7988033Z return func(*args, **kwargs) 2025-12-04T09:47:40.7988409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.7988840Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.7989298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.7989813Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.7990016Z 2025-12-04T09:47:40.7990120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7990479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7990802Z res = mod(**inputs) 2025-12-04T09:47:40.7991189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7991618Z outputs = self.model.decoder( 2025-12-04T09:47:40.7992045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7992473Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7992854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.7993254Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.7993656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7994060Z return func(*args, **kwargs) 2025-12-04T09:47:40.7994484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.7994912Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.7995342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.7995752Z return func(*args, **kwargs) 2025-12-04T09:47:40.7996169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.7996623Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.7996773Z 2025-12-04T09:47:40.7996885Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.7997271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.7997618Z res = mod(**inputs) 2025-12-04T09:47:40.7998013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.7998486Z outputs = self.model.decoder( 2025-12-04T09:47:40.7998918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.7999372Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.7999749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8000160Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8000653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8001107Z return func(*args, **kwargs) 2025-12-04T09:47:40.8001537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8002063Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8002256Z 2025-12-04T09:47:40.8002386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8002784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8003137Z res = mod(**inputs) 2025-12-04T09:47:40.8003544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8003981Z outputs = self.model.decoder( 2025-12-04T09:47:40.8004399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8004853Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8005235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8005624Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8006050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8006459Z return func(*args, **kwargs) 2025-12-04T09:47:40.8006880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8007345Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8007769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.8008144Z return self.act(input) 2025-12-04T09:47:40.8008267Z 2025-12-04T09:47:40.8008392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8008816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8009166Z res = mod(**inputs) 2025-12-04T09:47:40.8009569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8009994Z outputs = self.model.decoder( 2025-12-04T09:47:40.8010417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8010849Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8011227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8011597Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8011976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8012352Z return func(*args, **kwargs) 2025-12-04T09:47:40.8012728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.8013133Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.8013277Z 2025-12-04T09:47:40.8013379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8013761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8014079Z res = mod(**inputs) 2025-12-04T09:47:40.8014452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8014852Z outputs = self.model.decoder( 2025-12-04T09:47:40.8015234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8015645Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8015993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8016360Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8016744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8017122Z return func(*args, **kwargs) 2025-12-04T09:47:40.8017504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T09:47:40.8017908Z hidden_states = residual + hidden_states 2025-12-04T09:47:40.8018046Z 2025-12-04T09:47:40.8018150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8018511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8018881Z res = mod(**inputs) 2025-12-04T09:47:40.8019257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8019666Z outputs = self.model.decoder( 2025-12-04T09:47:40.8020071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8020489Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8020832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8021190Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8021565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8021926Z return func(*args, **kwargs) 2025-12-04T09:47:40.8022310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8022735Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8023135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8023503Z return func(*args, **kwargs) 2025-12-04T09:47:40.8023885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.8024357Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.8024558Z 2025-12-04T09:47:40.8024670Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8025020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8025340Z res = mod(**inputs) 2025-12-04T09:47:40.8025710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8026105Z outputs = self.model.decoder( 2025-12-04T09:47:40.8026506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8026916Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8027292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8027699Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8028112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8028516Z return func(*args, **kwargs) 2025-12-04T09:47:40.8028928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8029362Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8029822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8030222Z return func(*args, **kwargs) 2025-12-04T09:47:40.8030643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.8031080Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.8031233Z 2025-12-04T09:47:40.8031350Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8031737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8032078Z res = mod(**inputs) 2025-12-04T09:47:40.8032477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8032906Z outputs = self.model.decoder( 2025-12-04T09:47:40.8033339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8033773Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8034149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8034541Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8034944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8035345Z return func(*args, **kwargs) 2025-12-04T09:47:40.8035763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8036216Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8036640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8037047Z return func(*args, **kwargs) 2025-12-04T09:47:40.8037463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.8037900Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.8038061Z 2025-12-04T09:47:40.8038153Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.8038413Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8038804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8039147Z res = mod(**inputs) 2025-12-04T09:47:40.8039551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8039993Z outputs = self.model.decoder( 2025-12-04T09:47:40.8040415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8040935Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8041320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8041740Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8042181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8042594Z return func(*args, **kwargs) 2025-12-04T09:47:40.8043049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8043505Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8043930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8044321Z return func(*args, **kwargs) 2025-12-04T09:47:40.8044716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.8045157Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.8045610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.8046116Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.8046305Z 2025-12-04T09:47:40.8046426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8046796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8047291Z res = mod(**inputs) 2025-12-04T09:47:40.8047687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8048143Z outputs = self.model.decoder( 2025-12-04T09:47:40.8048593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8049000Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8049359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8049729Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8050129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8050520Z return func(*args, **kwargs) 2025-12-04T09:47:40.8050917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8051339Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8051753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8052140Z return func(*args, **kwargs) 2025-12-04T09:47:40.8052530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.8052946Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.8053091Z 2025-12-04T09:47:40.8053197Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8053561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8053885Z res = mod(**inputs) 2025-12-04T09:47:40.8054316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8054723Z outputs = self.model.decoder( 2025-12-04T09:47:40.8055123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8055578Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8055943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8056317Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8056709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8057083Z return func(*args, **kwargs) 2025-12-04T09:47:40.8057501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8057929Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8058095Z 2025-12-04T09:47:40.8058198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8058546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8058861Z res = mod(**inputs) 2025-12-04T09:47:40.8059223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8059654Z outputs = self.model.decoder( 2025-12-04T09:47:40.8060044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8060471Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8060813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8061171Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8061543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8061911Z return func(*args, **kwargs) 2025-12-04T09:47:40.8062281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8062724Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8063109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.8063438Z return self.act(input) 2025-12-04T09:47:40.8063553Z 2025-12-04T09:47:40.8063656Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8064006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8064318Z res = mod(**inputs) 2025-12-04T09:47:40.8064672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8065061Z outputs = self.model.decoder( 2025-12-04T09:47:40.8065442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8065822Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8066163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8066522Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8066894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8067253Z return func(*args, **kwargs) 2025-12-04T09:47:40.8067632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.8068025Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.8068158Z 2025-12-04T09:47:40.8068268Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8068607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8068918Z res = mod(**inputs) 2025-12-04T09:47:40.8069277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8069663Z outputs = self.model.decoder( 2025-12-04T09:47:40.8070041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8070431Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8070770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8071141Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8071518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8071884Z return func(*args, **kwargs) 2025-12-04T09:47:40.8072256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8072664Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8073072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8073437Z return func(*args, **kwargs) 2025-12-04T09:47:40.8073839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.8074299Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.8074504Z 2025-12-04T09:47:40.8074607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8074951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8075257Z res = mod(**inputs) 2025-12-04T09:47:40.8075615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8076004Z outputs = self.model.decoder( 2025-12-04T09:47:40.8076396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8076787Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8077138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8077507Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8077888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8078271Z return func(*args, **kwargs) 2025-12-04T09:47:40.8078665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8079097Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8079515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8079925Z return func(*args, **kwargs) 2025-12-04T09:47:40.8080340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.8080862Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.8081022Z 2025-12-04T09:47:40.8081143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8081548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8081909Z res = mod(**inputs) 2025-12-04T09:47:40.8082326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8082730Z outputs = self.model.decoder( 2025-12-04T09:47:40.8083123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8083519Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8083881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8084254Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8084643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8085017Z return func(*args, **kwargs) 2025-12-04T09:47:40.8085443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8085863Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8086262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8086628Z return func(*args, **kwargs) 2025-12-04T09:47:40.8087021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.8087467Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.8087611Z 2025-12-04T09:47:40.8087694Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.8087938Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8088318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8088654Z res = mod(**inputs) 2025-12-04T09:47:40.8089021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8089419Z outputs = self.model.decoder( 2025-12-04T09:47:40.8089807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8090196Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8090546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8090923Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8091299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8091661Z return func(*args, **kwargs) 2025-12-04T09:47:40.8092045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8092466Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8092864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8093227Z return func(*args, **kwargs) 2025-12-04T09:47:40.8093605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.8094031Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.8094478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.8094972Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.8095165Z 2025-12-04T09:47:40.8095274Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8095642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8095972Z res = mod(**inputs) 2025-12-04T09:47:40.8096342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8096745Z outputs = self.model.decoder( 2025-12-04T09:47:40.8097141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8097544Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8097899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8098271Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8098651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8099037Z return func(*args, **kwargs) 2025-12-04T09:47:40.8099451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8099878Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8100277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8100659Z return func(*args, **kwargs) 2025-12-04T09:47:40.8101046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.8101475Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.8101623Z 2025-12-04T09:47:40.8101731Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8102119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8102448Z res = mod(**inputs) 2025-12-04T09:47:40.8102824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8103238Z outputs = self.model.decoder( 2025-12-04T09:47:40.8103639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8104044Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8104399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8104798Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8105184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8105559Z return func(*args, **kwargs) 2025-12-04T09:47:40.8105954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8106403Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8106580Z 2025-12-04T09:47:40.8106693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8107049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8107378Z res = mod(**inputs) 2025-12-04T09:47:40.8107753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8108155Z outputs = self.model.decoder( 2025-12-04T09:47:40.8108555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8108960Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8109323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8109692Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8110083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8110467Z return func(*args, **kwargs) 2025-12-04T09:47:40.8110864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8111310Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8111714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.8112077Z return self.act(input) 2025-12-04T09:47:40.8112190Z 2025-12-04T09:47:40.8112298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8112670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8113000Z res = mod(**inputs) 2025-12-04T09:47:40.8113404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8113810Z outputs = self.model.decoder( 2025-12-04T09:47:40.8114208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8114626Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8114969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8115338Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8115736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8116115Z return func(*args, **kwargs) 2025-12-04T09:47:40.8116525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.8116942Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.8117084Z 2025-12-04T09:47:40.8117198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8117566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8117887Z res = mod(**inputs) 2025-12-04T09:47:40.8118263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8118697Z outputs = self.model.decoder( 2025-12-04T09:47:40.8119091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8119500Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8119859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8120234Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8120721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8121148Z return func(*args, **kwargs) 2025-12-04T09:47:40.8121581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T09:47:40.8122019Z hidden_states = residual + hidden_states 2025-12-04T09:47:40.8122173Z 2025-12-04T09:47:40.8122285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8122674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8123020Z res = mod(**inputs) 2025-12-04T09:47:40.8123412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8123848Z outputs = self.model.decoder( 2025-12-04T09:47:40.8124250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8124657Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8125008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8125379Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8125762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8126145Z return func(*args, **kwargs) 2025-12-04T09:47:40.8126562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8127014Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8127443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8127837Z return func(*args, **kwargs) 2025-12-04T09:47:40.8128280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.8128797Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.8129014Z 2025-12-04T09:47:40.8129138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8129518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8129865Z res = mod(**inputs) 2025-12-04T09:47:40.8130303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8130731Z outputs = self.model.decoder( 2025-12-04T09:47:40.8131174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8131605Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8131984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8132371Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8132778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8133180Z return func(*args, **kwargs) 2025-12-04T09:47:40.8133588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8134061Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8134490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8134892Z return func(*args, **kwargs) 2025-12-04T09:47:40.8135301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.8135736Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.8135880Z 2025-12-04T09:47:40.8136000Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8136383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8136723Z res = mod(**inputs) 2025-12-04T09:47:40.8137121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8137554Z outputs = self.model.decoder( 2025-12-04T09:47:40.8137971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8138401Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8138779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8139174Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8139576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8139980Z return func(*args, **kwargs) 2025-12-04T09:47:40.8140391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8140837Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8141267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8141670Z return func(*args, **kwargs) 2025-12-04T09:47:40.8142085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.8142525Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.8142682Z 2025-12-04T09:47:40.8142775Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.8143059Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8143448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8143797Z res = mod(**inputs) 2025-12-04T09:47:40.8144198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8144629Z outputs = self.model.decoder( 2025-12-04T09:47:40.8145049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8145510Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8145890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8146307Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8146713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8147338Z return func(*args, **kwargs) 2025-12-04T09:47:40.8147750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8148182Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8148606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8149056Z return func(*args, **kwargs) 2025-12-04T09:47:40.8149452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.8149891Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.8150381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.8150883Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.8151072Z 2025-12-04T09:47:40.8151189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8151555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8151894Z res = mod(**inputs) 2025-12-04T09:47:40.8152278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8152686Z outputs = self.model.decoder( 2025-12-04T09:47:40.8153091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8153503Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8153868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8154239Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8154630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8155010Z return func(*args, **kwargs) 2025-12-04T09:47:40.8155396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8155823Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8156238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8156636Z return func(*args, **kwargs) 2025-12-04T09:47:40.8157044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.8157486Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.8157640Z 2025-12-04T09:47:40.8157753Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8158170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8158519Z res = mod(**inputs) 2025-12-04T09:47:40.8158925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8159366Z outputs = self.model.decoder( 2025-12-04T09:47:40.8159793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8160256Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8160698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8161119Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8161560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8161979Z return func(*args, **kwargs) 2025-12-04T09:47:40.8162409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8162862Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8163038Z 2025-12-04T09:47:40.8163145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8163509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8163860Z res = mod(**inputs) 2025-12-04T09:47:40.8164234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8164651Z outputs = self.model.decoder( 2025-12-04T09:47:40.8165057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8165467Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8165817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8166188Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8166576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8166958Z return func(*args, **kwargs) 2025-12-04T09:47:40.8167343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8167829Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8168302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.8168715Z return self.act(input) 2025-12-04T09:47:40.8168860Z 2025-12-04T09:47:40.8169020Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8169382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8169703Z res = mod(**inputs) 2025-12-04T09:47:40.8169971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8170046Z outputs = self.model.decoder( 2025-12-04T09:47:40.8170314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8170389Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8170609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8170698Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8170940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8171035Z return func(*args, **kwargs) 2025-12-04T09:47:40.8171297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.8171378Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.8171382Z 2025-12-04T09:47:40.8171491Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8171685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8171757Z res = mod(**inputs) 2025-12-04T09:47:40.8172042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8172115Z outputs = self.model.decoder( 2025-12-04T09:47:40.8172401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8172474Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8172696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8172784Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8173024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8173100Z return func(*args, **kwargs) 2025-12-04T09:47:40.8173361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8173512Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8173760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8173830Z return func(*args, **kwargs) 2025-12-04T09:47:40.8174089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.8174246Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.8174249Z 2025-12-04T09:47:40.8174353Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8174557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8174621Z res = mod(**inputs) 2025-12-04T09:47:40.8174886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8174970Z outputs = self.model.decoder( 2025-12-04T09:47:40.8175232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8175310Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8175532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8175611Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8175860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8175928Z return func(*args, **kwargs) 2025-12-04T09:47:40.8176186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8176295Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8176536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8176612Z return func(*args, **kwargs) 2025-12-04T09:47:40.8176874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.8176956Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.8176960Z 2025-12-04T09:47:40.8177096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8177291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8177361Z res = mod(**inputs) 2025-12-04T09:47:40.8177620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8177693Z outputs = self.model.decoder( 2025-12-04T09:47:40.8177972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8178064Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8178293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8178395Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8178646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8178723Z return func(*args, **kwargs) 2025-12-04T09:47:40.8178989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8179091Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8179342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8179428Z return func(*args, **kwargs) 2025-12-04T09:47:40.8179702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.8179794Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.8179798Z 2025-12-04T09:47:40.8179879Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.8179988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8180183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8180249Z res = mod(**inputs) 2025-12-04T09:47:40.8180528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8180603Z outputs = self.model.decoder( 2025-12-04T09:47:40.8180878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8180954Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8181182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8181270Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8181520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8181592Z return func(*args, **kwargs) 2025-12-04T09:47:40.8181872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8181970Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8182217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8182284Z return func(*args, **kwargs) 2025-12-04T09:47:40.8182544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.8182655Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.8182942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.8183081Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.8183085Z 2025-12-04T09:47:40.8183204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8183402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8183475Z res = mod(**inputs) 2025-12-04T09:47:40.8183736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8183811Z outputs = self.model.decoder( 2025-12-04T09:47:40.8184082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8184169Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8184396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8184495Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8184739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8184823Z return func(*args, **kwargs) 2025-12-04T09:47:40.8185084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8185187Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8185425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8185509Z return func(*args, **kwargs) 2025-12-04T09:47:40.8185791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.8185870Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.8185873Z 2025-12-04T09:47:40.8185972Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8186170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8186233Z res = mod(**inputs) 2025-12-04T09:47:40.8186498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8186571Z outputs = self.model.decoder( 2025-12-04T09:47:40.8186830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8186908Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8187128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8187205Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8187445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8187513Z return func(*args, **kwargs) 2025-12-04T09:47:40.8187774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8187888Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8187892Z 2025-12-04T09:47:40.8188076Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8188272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8188345Z res = mod(**inputs) 2025-12-04T09:47:40.8188611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8188687Z outputs = self.model.decoder( 2025-12-04T09:47:40.8188962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8189036Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8189287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8189370Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8189618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8189696Z return func(*args, **kwargs) 2025-12-04T09:47:40.8189963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8190092Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8190967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.8191035Z return self.act(input) 2025-12-04T09:47:40.8191038Z 2025-12-04T09:47:40.8191143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8191350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8191416Z res = mod(**inputs) 2025-12-04T09:47:40.8191680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8191750Z outputs = self.model.decoder( 2025-12-04T09:47:40.8192014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8192086Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8192321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8192410Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8192648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8192725Z return func(*args, **kwargs) 2025-12-04T09:47:40.8192984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.8193063Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.8193067Z 2025-12-04T09:47:40.8193175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8193366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8193430Z res = mod(**inputs) 2025-12-04T09:47:40.8193696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8193771Z outputs = self.model.decoder( 2025-12-04T09:47:40.8194038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8194110Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8194330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8194418Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8194664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8194732Z return func(*args, **kwargs) 2025-12-04T09:47:40.8195002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T09:47:40.8195084Z hidden_states = residual + hidden_states 2025-12-04T09:47:40.8195089Z 2025-12-04T09:47:40.8195201Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8195403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8195469Z res = mod(**inputs) 2025-12-04T09:47:40.8195743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8195816Z outputs = self.model.decoder( 2025-12-04T09:47:40.8196105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8196183Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8196410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8196497Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8196749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8196835Z return func(*args, **kwargs) 2025-12-04T09:47:40.8197119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8197235Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8197499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8197573Z return func(*args, **kwargs) 2025-12-04T09:47:40.8197856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:47:40.8198025Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:47:40.8198029Z 2025-12-04T09:47:40.8198140Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8198384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8198455Z res = mod(**inputs) 2025-12-04T09:47:40.8198737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8198825Z outputs = self.model.decoder( 2025-12-04T09:47:40.8199115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8199191Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8199443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8199529Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8199806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8199881Z return func(*args, **kwargs) 2025-12-04T09:47:40.8200173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8200291Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8200636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8200727Z return func(*args, **kwargs) 2025-12-04T09:47:40.8201024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:47:40.8201115Z key_states = self.k_proj(current_states) 2025-12-04T09:47:40.8201119Z 2025-12-04T09:47:40.8201243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8201462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8201533Z res = mod(**inputs) 2025-12-04T09:47:40.8201836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8201919Z outputs = self.model.decoder( 2025-12-04T09:47:40.8202195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8202268Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8202503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8202590Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8202823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8202889Z return func(*args, **kwargs) 2025-12-04T09:47:40.8203174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8203284Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8203578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8203654Z return func(*args, **kwargs) 2025-12-04T09:47:40.8203963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:47:40.8204073Z value_states = self.v_proj(current_states) 2025-12-04T09:47:40.8204077Z 2025-12-04T09:47:40.8204167Z cudagraph partition due to non gpu ops 2025-12-04T09:47:40.8204287Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8204505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8204576Z res = mod(**inputs) 2025-12-04T09:47:40.8204879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8204980Z outputs = self.model.decoder( 2025-12-04T09:47:40.8205273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8205363Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8205611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8205705Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8205973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8206048Z return func(*args, **kwargs) 2025-12-04T09:47:40.8206342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8206449Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8206717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8206801Z return func(*args, **kwargs) 2025-12-04T09:47:40.8207093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:47:40.8207212Z attn_output, attn_weights = attention_interface( 2025-12-04T09:47:40.8207537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:47:40.8207687Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:47:40.8207691Z 2025-12-04T09:47:40.8207815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8208035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8208114Z res = mod(**inputs) 2025-12-04T09:47:40.8208408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8208490Z outputs = self.model.decoder( 2025-12-04T09:47:40.8208790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8208872Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8209138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8209238Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8209513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8209588Z return func(*args, **kwargs) 2025-12-04T09:47:40.8209855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:47:40.8209956Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:40.8210227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8210296Z return func(*args, **kwargs) 2025-12-04T09:47:40.8210585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:47:40.8210671Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:40.8210676Z 2025-12-04T09:47:40.8210783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8210992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8211058Z res = mod(**inputs) 2025-12-04T09:47:40.8211326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8211409Z outputs = self.model.decoder( 2025-12-04T09:47:40.8211693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8211776Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8212005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8212085Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8212345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8212415Z return func(*args, **kwargs) 2025-12-04T09:47:40.8212688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8212812Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8212815Z 2025-12-04T09:47:40.8212921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8213133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8213200Z res = mod(**inputs) 2025-12-04T09:47:40.8213467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8213550Z outputs = self.model.decoder( 2025-12-04T09:47:40.8213818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8213899Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8214126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8214207Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8214460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8214533Z return func(*args, **kwargs) 2025-12-04T09:47:40.8214808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:47:40.8214928Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:40.8215150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:40.8215228Z return self.act(input) 2025-12-04T09:47:40.8215249Z 2025-12-04T09:47:40.8215355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8215558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8215632Z res = mod(**inputs) 2025-12-04T09:47:40.8215901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:47:40.8215982Z outputs = self.model.decoder( 2025-12-04T09:47:40.8216254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:47:40.8216345Z layer_outputs = decoder_layer( 2025-12-04T09:47:40.8216584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:40.8216681Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:40.8216931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:40.8217011Z return func(*args, **kwargs) 2025-12-04T09:47:40.8217282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:47:40.8217373Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:40.8217376Z 2025-12-04T09:47:40.8217481Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8217697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8217772Z res = mod(**inputs) 2025-12-04T09:47:40.8218040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1649, in forward 2025-12-04T09:47:40.8218129Z logits = self.lm_head(outputs[0]) 2025-12-04T09:47:40.8218133Z 2025-12-04T09:47:40.8218238Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:40.8218439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:40.8218510Z res = mod(**inputs) 2025-12-04T09:47:40.8218780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1655, in forward 2025-12-04T09:47:40.8218931Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:47:40.8218942Z 2025-12-04T09:47:51.2981702Z Compilation time (from dynamo_timed): 17.558680439 2025-12-04T09:47:51.3000947Z pass 2025-12-04T09:47:51.3001897Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:51.3002841Z TIMING: _recursive_pre_grad_passes:0.00792 _recursive_joint_graph_passes:0.71719 _recursive_post_grad_passes:0.07048 async_compile.wait:0.84095 code_gen:10.26607 inductor_compile:11.67082 backend_compile:15.04769 gc:0.00106 entire_frame_compile:17.55868 total_wall_time:17.55868 2025-12-04T09:47:51.3003895Z STATS: call_* op count: 333 | FakeTensorMode.__torch_dispatch__:7342 | FakeTensor.__torch_dispatch__:4382 | ProxyTorchDispatchMode.__torch_dispatch__:2031 2025-12-04T09:47:51.3004449Z Dynamo produced 1 graphs covering 333 ops with 0 graph breaks (0 unique) 2025-12-04T09:47:53.7362085Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:47:53.7363004Z import pynvml # type: ignore[import] 2025-12-04T09:47:57.0361658Z 2025-12-04T09:47:57.0481757Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-12-04T09:47:57.0486361Z WARNING:transformers.models.roberta.modeling_roberta:If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-12-04T09:47:58.2020180Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-12-04T09:47:58.2021150Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-12-04T09:47:58.2022126Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-12-04T09:47:58.2023240Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-12-04T09:47:58.3923116Z 2025-12-04T09:47:58.3925297Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:47:58.3938692Z cpu eval RobertaForCausalLM 2025-12-04T09:47:58.9753182Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:59.2595778Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:59.5400368Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:06.8067778Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8068736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8069080Z res = mod(**inputs) 2025-12-04T09:48:06.8069505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8069932Z outputs = self.roberta( 2025-12-04T09:48:06.8070332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 798, in forward 2025-12-04T09:48:06.8070744Z embedding_output = self.embeddings( 2025-12-04T09:48:06.8071161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 87, in forward 2025-12-04T09:48:06.8071712Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-12-04T09:48:06.8072328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1548, in create_position_ids_from_input_ids 2025-12-04T09:48:06.8072817Z mask = input_ids.ne(padding_idx).int() 2025-12-04T09:48:06.8072970Z 2025-12-04T09:48:06.8073058Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8073278Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8073482Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8073695Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8073907Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8074113Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8074315Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8074558Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8074766Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8074974Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8075182Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8075383Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8075626Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8076012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8076357Z res = mod(**inputs) 2025-12-04T09:48:06.8076762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8077197Z outputs = self.roberta( 2025-12-04T09:48:06.8077740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 798, in forward 2025-12-04T09:48:06.8078177Z embedding_output = self.embeddings( 2025-12-04T09:48:06.8078615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 87, in forward 2025-12-04T09:48:06.8079181Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-12-04T09:48:06.8079819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1549, in create_position_ids_from_input_ids 2025-12-04T09:48:06.8080725Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:48:06.8081016Z 2025-12-04T09:48:06.8090566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8091247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8091630Z res = mod(**inputs) 2025-12-04T09:48:06.8092069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8092522Z outputs = self.roberta( 2025-12-04T09:48:06.8092948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 798, in forward 2025-12-04T09:48:06.8093391Z embedding_output = self.embeddings( 2025-12-04T09:48:06.8093877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 87, in forward 2025-12-04T09:48:06.8094458Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-12-04T09:48:06.8095109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1549, in create_position_ids_from_input_ids 2025-12-04T09:48:06.8095735Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:48:06.8096007Z 2025-12-04T09:48:06.8096130Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8096532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8096887Z res = mod(**inputs) 2025-12-04T09:48:06.8097286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8097725Z outputs = self.roberta( 2025-12-04T09:48:06.8098152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8098591Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8099018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8099456Z layer_outputs = layer_module( 2025-12-04T09:48:06.8099852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8100251Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8100683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8101096Z return func(*args, **kwargs) 2025-12-04T09:48:06.8101517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8101958Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8102381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8102794Z return func(*args, **kwargs) 2025-12-04T09:48:06.8103213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8103608Z self_outputs = self.self( 2025-12-04T09:48:06.8103983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8104369Z return func(*args, **kwargs) 2025-12-04T09:48:06.8104757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8105312Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8105610Z 2025-12-04T09:48:06.8105721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8106098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8106439Z res = mod(**inputs) 2025-12-04T09:48:06.8106825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8107241Z outputs = self.roberta( 2025-12-04T09:48:06.8107628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8108030Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8108442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8109715Z layer_outputs = layer_module( 2025-12-04T09:48:06.8110102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8110483Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8110879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8111263Z return func(*args, **kwargs) 2025-12-04T09:48:06.8111646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8112057Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8112449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8112829Z return func(*args, **kwargs) 2025-12-04T09:48:06.8113218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8113623Z self_outputs = self.self( 2025-12-04T09:48:06.8114008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8114402Z return func(*args, **kwargs) 2025-12-04T09:48:06.8114811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8115229Z self.key(current_states) 2025-12-04T09:48:06.8115352Z 2025-12-04T09:48:06.8115474Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8115859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8116200Z res = mod(**inputs) 2025-12-04T09:48:06.8116598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8117012Z outputs = self.roberta( 2025-12-04T09:48:06.8117419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8117841Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8118263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8118673Z layer_outputs = layer_module( 2025-12-04T09:48:06.8119074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8119471Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8119879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8120284Z return func(*args, **kwargs) 2025-12-04T09:48:06.8120799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8121270Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8121680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8122077Z return func(*args, **kwargs) 2025-12-04T09:48:06.8122478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8122872Z self_outputs = self.self( 2025-12-04T09:48:06.8123250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8123628Z return func(*args, **kwargs) 2025-12-04T09:48:06.8124014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8124409Z self.value(current_states) 2025-12-04T09:48:06.8124559Z 2025-12-04T09:48:06.8124646Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8124902Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8125269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8125590Z res = mod(**inputs) 2025-12-04T09:48:06.8125971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8126376Z outputs = self.roberta( 2025-12-04T09:48:06.8126752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8127155Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8127553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8127953Z layer_outputs = layer_module( 2025-12-04T09:48:06.8128307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8128681Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8129069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8129446Z return func(*args, **kwargs) 2025-12-04T09:48:06.8129830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8130240Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8130634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8131010Z return func(*args, **kwargs) 2025-12-04T09:48:06.8131398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8131798Z self_outputs = self.self( 2025-12-04T09:48:06.8132172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8132544Z return func(*args, **kwargs) 2025-12-04T09:48:06.8132929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8133404Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8133601Z 2025-12-04T09:48:06.8133716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8134102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8134454Z res = mod(**inputs) 2025-12-04T09:48:06.8134857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8135246Z outputs = self.roberta( 2025-12-04T09:48:06.8135631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8136048Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8136462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8136855Z layer_outputs = layer_module( 2025-12-04T09:48:06.8137215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8137588Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8137970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8138346Z return func(*args, **kwargs) 2025-12-04T09:48:06.8138733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8139165Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8139550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8139943Z return func(*args, **kwargs) 2025-12-04T09:48:06.8140353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8140834Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8141306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8141720Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8141864Z 2025-12-04T09:48:06.8141981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8142344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8142678Z res = mod(**inputs) 2025-12-04T09:48:06.8143056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8143453Z outputs = self.roberta( 2025-12-04T09:48:06.8143835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8144240Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8144638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8145029Z layer_outputs = layer_module( 2025-12-04T09:48:06.8145388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8145763Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8146150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8146527Z return func(*args, **kwargs) 2025-12-04T09:48:06.8146919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8147541Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8148024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8148435Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8148876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8149367Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8149817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8150262Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8150411Z 2025-12-04T09:48:06.8150520Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8150889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8151240Z res = mod(**inputs) 2025-12-04T09:48:06.8151624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8152030Z outputs = self.roberta( 2025-12-04T09:48:06.8152415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8152821Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8153225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8153656Z layer_outputs = layer_module( 2025-12-04T09:48:06.8154009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8154381Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8154772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8155172Z return func(*args, **kwargs) 2025-12-04T09:48:06.8155557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8155966Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8156370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8156766Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8157225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8157745Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8158218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8158678Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8159096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8159467Z return self.act(input) 2025-12-04T09:48:06.8159589Z 2025-12-04T09:48:06.8159712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8160135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8160552Z res = mod(**inputs) 2025-12-04T09:48:06.8160960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8161395Z outputs = self.roberta( 2025-12-04T09:48:06.8161833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8162289Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8162743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8163186Z layer_outputs = layer_module( 2025-12-04T09:48:06.8163566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8163961Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8164365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8164773Z return func(*args, **kwargs) 2025-12-04T09:48:06.8165189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8165640Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8166062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8166501Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8166968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8167495Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8167983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8168426Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8168577Z 2025-12-04T09:48:06.8168700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8169105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8169446Z res = mod(**inputs) 2025-12-04T09:48:06.8169839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8170215Z outputs = self.roberta( 2025-12-04T09:48:06.8170589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8170990Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8171407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8171827Z layer_outputs = layer_module( 2025-12-04T09:48:06.8172554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8172936Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8173315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8173689Z return func(*args, **kwargs) 2025-12-04T09:48:06.8174064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8174468Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8174850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8175211Z return func(*args, **kwargs) 2025-12-04T09:48:06.8175598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8176006Z self_outputs = self.self( 2025-12-04T09:48:06.8176370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8176736Z return func(*args, **kwargs) 2025-12-04T09:48:06.8177113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8177654Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8177926Z 2025-12-04T09:48:06.8178059Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8178416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8178736Z res = mod(**inputs) 2025-12-04T09:48:06.8179106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8179487Z outputs = self.roberta( 2025-12-04T09:48:06.8179876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8180300Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8180702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8181108Z layer_outputs = layer_module( 2025-12-04T09:48:06.8181508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8181894Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8182273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8182654Z return func(*args, **kwargs) 2025-12-04T09:48:06.8183043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8183454Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8183865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8184255Z return func(*args, **kwargs) 2025-12-04T09:48:06.8184636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8185019Z self_outputs = self.self( 2025-12-04T09:48:06.8185376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8185745Z return func(*args, **kwargs) 2025-12-04T09:48:06.8186122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8186503Z self.key(current_states) 2025-12-04T09:48:06.8186625Z 2025-12-04T09:48:06.8186731Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8187089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8187408Z res = mod(**inputs) 2025-12-04T09:48:06.8187765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8188150Z outputs = self.roberta( 2025-12-04T09:48:06.8188526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8188910Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8189293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8189678Z layer_outputs = layer_module( 2025-12-04T09:48:06.8190024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8190383Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8190770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8191170Z return func(*args, **kwargs) 2025-12-04T09:48:06.8191562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8191986Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8192411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8192808Z return func(*args, **kwargs) 2025-12-04T09:48:06.8193203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8193660Z self_outputs = self.self( 2025-12-04T09:48:06.8194052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8194471Z return func(*args, **kwargs) 2025-12-04T09:48:06.8194873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8195283Z self.value(current_states) 2025-12-04T09:48:06.8195404Z 2025-12-04T09:48:06.8195516Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8195761Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8196152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8196506Z res = mod(**inputs) 2025-12-04T09:48:06.8196881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8197281Z outputs = self.roberta( 2025-12-04T09:48:06.8197673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8198122Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8198517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8198914Z layer_outputs = layer_module( 2025-12-04T09:48:06.8199276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8199652Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8200036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8200501Z return func(*args, **kwargs) 2025-12-04T09:48:06.8200929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8201363Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8201773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8202175Z return func(*args, **kwargs) 2025-12-04T09:48:06.8202554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8202936Z self_outputs = self.self( 2025-12-04T09:48:06.8203299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8203668Z return func(*args, **kwargs) 2025-12-04T09:48:06.8204043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8204484Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8204672Z 2025-12-04T09:48:06.8204777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8205137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8205449Z res = mod(**inputs) 2025-12-04T09:48:06.8205813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8206197Z outputs = self.roberta( 2025-12-04T09:48:06.8206570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8206973Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8207368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8207763Z layer_outputs = layer_module( 2025-12-04T09:48:06.8208116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8208483Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8208860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8209245Z return func(*args, **kwargs) 2025-12-04T09:48:06.8209617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8210038Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8210436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8210829Z return func(*args, **kwargs) 2025-12-04T09:48:06.8211234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8211715Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8212190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8212605Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8212752Z 2025-12-04T09:48:06.8212856Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8213215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8213543Z res = mod(**inputs) 2025-12-04T09:48:06.8213903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8214292Z outputs = self.roberta( 2025-12-04T09:48:06.8214666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8215056Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8215435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8215824Z layer_outputs = layer_module( 2025-12-04T09:48:06.8216174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8216531Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8216916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8217291Z return func(*args, **kwargs) 2025-12-04T09:48:06.8217675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8218063Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8218456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8218838Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8219241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8219719Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8220146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8220549Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8220687Z 2025-12-04T09:48:06.8220807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8221172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8221499Z res = mod(**inputs) 2025-12-04T09:48:06.8221870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8222245Z outputs = self.roberta( 2025-12-04T09:48:06.8222616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8223020Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8223388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8223771Z layer_outputs = layer_module( 2025-12-04T09:48:06.8224123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8224480Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8224842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8225200Z return func(*args, **kwargs) 2025-12-04T09:48:06.8225570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8225964Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8226364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8226741Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8227153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8227601Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8228026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8228445Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8228814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8229145Z return self.act(input) 2025-12-04T09:48:06.8229261Z 2025-12-04T09:48:06.8229365Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8229718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8230035Z res = mod(**inputs) 2025-12-04T09:48:06.8230385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8230760Z outputs = self.roberta( 2025-12-04T09:48:06.8231123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8231515Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8231901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8232281Z layer_outputs = layer_module( 2025-12-04T09:48:06.8232628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8233000Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8233388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8233753Z return func(*args, **kwargs) 2025-12-04T09:48:06.8234123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8234533Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8234926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8235321Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8235745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8236222Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8236663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8237077Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8237212Z 2025-12-04T09:48:06.8237323Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8237686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8238014Z res = mod(**inputs) 2025-12-04T09:48:06.8238376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8238760Z outputs = self.roberta( 2025-12-04T09:48:06.8239128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8239522Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8239943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8240339Z layer_outputs = layer_module( 2025-12-04T09:48:06.8240793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8241213Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8241647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8242073Z return func(*args, **kwargs) 2025-12-04T09:48:06.8242458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8242867Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8243256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8243629Z return func(*args, **kwargs) 2025-12-04T09:48:06.8244018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8244413Z self_outputs = self.self( 2025-12-04T09:48:06.8244773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8245149Z return func(*args, **kwargs) 2025-12-04T09:48:06.8245532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8246068Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8246329Z 2025-12-04T09:48:06.8246436Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8246801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8247245Z res = mod(**inputs) 2025-12-04T09:48:06.8247628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8248010Z outputs = self.roberta( 2025-12-04T09:48:06.8248391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8248792Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8249217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8249607Z layer_outputs = layer_module( 2025-12-04T09:48:06.8249953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8250311Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8250681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8251077Z return func(*args, **kwargs) 2025-12-04T09:48:06.8251460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8251853Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8252279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8252641Z return func(*args, **kwargs) 2025-12-04T09:48:06.8253011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8253388Z self_outputs = self.self( 2025-12-04T09:48:06.8253742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8254104Z return func(*args, **kwargs) 2025-12-04T09:48:06.8254494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8254874Z self.key(current_states) 2025-12-04T09:48:06.8254993Z 2025-12-04T09:48:06.8255096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8255444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8255746Z res = mod(**inputs) 2025-12-04T09:48:06.8256105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8256482Z outputs = self.roberta( 2025-12-04T09:48:06.8256842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8257214Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8257589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8257974Z layer_outputs = layer_module( 2025-12-04T09:48:06.8258310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8258658Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8259027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8259387Z return func(*args, **kwargs) 2025-12-04T09:48:06.8259746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8260134Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8260509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8260866Z return func(*args, **kwargs) 2025-12-04T09:48:06.8261236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8261626Z self_outputs = self.self( 2025-12-04T09:48:06.8261988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8262343Z return func(*args, **kwargs) 2025-12-04T09:48:06.8262723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8263102Z self.value(current_states) 2025-12-04T09:48:06.8263217Z 2025-12-04T09:48:06.8263304Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8263533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8263883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8264193Z res = mod(**inputs) 2025-12-04T09:48:06.8264546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8264945Z outputs = self.roberta( 2025-12-04T09:48:06.8265316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8265733Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8266107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8266486Z layer_outputs = layer_module( 2025-12-04T09:48:06.8266824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8267171Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8267539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8267918Z return func(*args, **kwargs) 2025-12-04T09:48:06.8268287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8268672Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8269048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8269410Z return func(*args, **kwargs) 2025-12-04T09:48:06.8269781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8270180Z self_outputs = self.self( 2025-12-04T09:48:06.8270529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8270888Z return func(*args, **kwargs) 2025-12-04T09:48:06.8271252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8271690Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8271873Z 2025-12-04T09:48:06.8271974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8272324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8272630Z res = mod(**inputs) 2025-12-04T09:48:06.8272985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8273360Z outputs = self.roberta( 2025-12-04T09:48:06.8273711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8274089Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8274462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8274838Z layer_outputs = layer_module( 2025-12-04T09:48:06.8275166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8275513Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8275882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8276260Z return func(*args, **kwargs) 2025-12-04T09:48:06.8276626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8277016Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8277387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8277743Z return func(*args, **kwargs) 2025-12-04T09:48:06.8278115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8278579Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8279045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8279440Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8279585Z 2025-12-04T09:48:06.8279691Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8280054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8280374Z res = mod(**inputs) 2025-12-04T09:48:06.8280820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8281237Z outputs = self.roberta( 2025-12-04T09:48:06.8281660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8282096Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8282492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8282893Z layer_outputs = layer_module( 2025-12-04T09:48:06.8283265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8283620Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8283998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8284369Z return func(*args, **kwargs) 2025-12-04T09:48:06.8284740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8285142Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8285540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8285929Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8286341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8286809Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8287245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8287643Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8287780Z 2025-12-04T09:48:06.8287882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8288234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8288554Z res = mod(**inputs) 2025-12-04T09:48:06.8288913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8289296Z outputs = self.roberta( 2025-12-04T09:48:06.8289673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8290060Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8290491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8290884Z layer_outputs = layer_module( 2025-12-04T09:48:06.8291233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8291588Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8291970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8292359Z return func(*args, **kwargs) 2025-12-04T09:48:06.8292736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8293133Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8293535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8293920Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8294338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8294799Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8295219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8295652Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8296038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8296403Z return self.act(input) 2025-12-04T09:48:06.8296522Z 2025-12-04T09:48:06.8296625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8296988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8297307Z res = mod(**inputs) 2025-12-04T09:48:06.8297673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8298075Z outputs = self.roberta( 2025-12-04T09:48:06.8298449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8298837Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8299229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8299621Z layer_outputs = layer_module( 2025-12-04T09:48:06.8299982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8300369Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8300770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8301161Z return func(*args, **kwargs) 2025-12-04T09:48:06.8301553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8301975Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8302386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8302793Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8303220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8303703Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8304167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8304579Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8304723Z 2025-12-04T09:48:06.8304827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8305180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8305496Z res = mod(**inputs) 2025-12-04T09:48:06.8305853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8306237Z outputs = self.roberta( 2025-12-04T09:48:06.8306638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8307039Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8307439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8307822Z layer_outputs = layer_module( 2025-12-04T09:48:06.8308167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8308518Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8308894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8309261Z return func(*args, **kwargs) 2025-12-04T09:48:06.8309632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8310063Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8310438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8310800Z return func(*args, **kwargs) 2025-12-04T09:48:06.8311166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8311546Z self_outputs = self.self( 2025-12-04T09:48:06.8311901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8312260Z return func(*args, **kwargs) 2025-12-04T09:48:06.8312623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8313133Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8313388Z 2025-12-04T09:48:06.8313497Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8313850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8314158Z res = mod(**inputs) 2025-12-04T09:48:06.8314520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8314899Z outputs = self.roberta( 2025-12-04T09:48:06.8315261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8315644Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8316026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8316409Z layer_outputs = layer_module( 2025-12-04T09:48:06.8316748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8317104Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8317478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8317839Z return func(*args, **kwargs) 2025-12-04T09:48:06.8318246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8318684Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8319099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8319493Z return func(*args, **kwargs) 2025-12-04T09:48:06.8319905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8320348Z self_outputs = self.self( 2025-12-04T09:48:06.8320815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8321221Z return func(*args, **kwargs) 2025-12-04T09:48:06.8321670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8322063Z self.key(current_states) 2025-12-04T09:48:06.8322179Z 2025-12-04T09:48:06.8322285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8322647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8322968Z res = mod(**inputs) 2025-12-04T09:48:06.8323332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8323734Z outputs = self.roberta( 2025-12-04T09:48:06.8324098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8324480Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8324851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8325227Z layer_outputs = layer_module( 2025-12-04T09:48:06.8325564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8325913Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8326270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8326629Z return func(*args, **kwargs) 2025-12-04T09:48:06.8326993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8327374Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8327744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8328107Z return func(*args, **kwargs) 2025-12-04T09:48:06.8328487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8328871Z self_outputs = self.self( 2025-12-04T09:48:06.8329235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8329612Z return func(*args, **kwargs) 2025-12-04T09:48:06.8330000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8330381Z self.value(current_states) 2025-12-04T09:48:06.8330503Z 2025-12-04T09:48:06.8330583Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8330823Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8331171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8331496Z res = mod(**inputs) 2025-12-04T09:48:06.8331872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8332282Z outputs = self.roberta( 2025-12-04T09:48:06.8332663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8333063Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8333459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8333851Z layer_outputs = layer_module( 2025-12-04T09:48:06.8334213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8334607Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8334990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8335376Z return func(*args, **kwargs) 2025-12-04T09:48:06.8335768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8336176Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8336555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8336939Z return func(*args, **kwargs) 2025-12-04T09:48:06.8337326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8337739Z self_outputs = self.self( 2025-12-04T09:48:06.8338104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8338504Z return func(*args, **kwargs) 2025-12-04T09:48:06.8338918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8339404Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8339604Z 2025-12-04T09:48:06.8339717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8340103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8340433Z res = mod(**inputs) 2025-12-04T09:48:06.8340800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8341198Z outputs = self.roberta( 2025-12-04T09:48:06.8341582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8341990Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8342380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8342780Z layer_outputs = layer_module( 2025-12-04T09:48:06.8343136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8343498Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8343884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8344261Z return func(*args, **kwargs) 2025-12-04T09:48:06.8344650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8345054Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8345444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8345821Z return func(*args, **kwargs) 2025-12-04T09:48:06.8346206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8346673Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8347313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8347733Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8347875Z 2025-12-04T09:48:06.8347983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8348355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8348730Z res = mod(**inputs) 2025-12-04T09:48:06.8349104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8349495Z outputs = self.roberta( 2025-12-04T09:48:06.8349900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8350307Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8350709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8351113Z layer_outputs = layer_module( 2025-12-04T09:48:06.8351463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8351827Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8352199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8352595Z return func(*args, **kwargs) 2025-12-04T09:48:06.8352980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8353385Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8353783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8354177Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8354600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8355075Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8355509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8355917Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8356054Z 2025-12-04T09:48:06.8356166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8356517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8356839Z res = mod(**inputs) 2025-12-04T09:48:06.8357212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8357601Z outputs = self.roberta( 2025-12-04T09:48:06.8357963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8358352Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8358751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8359144Z layer_outputs = layer_module( 2025-12-04T09:48:06.8359506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8359879Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8360274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8360708Z return func(*args, **kwargs) 2025-12-04T09:48:06.8361143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8361555Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8361965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8362359Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8362795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8363299Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8363722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8364163Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8364553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8364901Z return self.act(input) 2025-12-04T09:48:06.8365017Z 2025-12-04T09:48:06.8365122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8365488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8365807Z res = mod(**inputs) 2025-12-04T09:48:06.8366171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8366574Z outputs = self.roberta( 2025-12-04T09:48:06.8366946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8367334Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8367711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8368099Z layer_outputs = layer_module( 2025-12-04T09:48:06.8368446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8368806Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8369181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8369603Z return func(*args, **kwargs) 2025-12-04T09:48:06.8369992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8370404Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8370796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8371177Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8371586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8372047Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8372506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8372917Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8373057Z 2025-12-04T09:48:06.8373173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8373540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8373877Z res = mod(**inputs) 2025-12-04T09:48:06.8374242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8374625Z outputs = self.roberta( 2025-12-04T09:48:06.8375002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8375387Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8375770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8376153Z layer_outputs = layer_module( 2025-12-04T09:48:06.8376501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8376868Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8377254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8377628Z return func(*args, **kwargs) 2025-12-04T09:48:06.8378063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8378460Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8378828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8379190Z return func(*args, **kwargs) 2025-12-04T09:48:06.8379560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8379941Z self_outputs = self.self( 2025-12-04T09:48:06.8380286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8380660Z return func(*args, **kwargs) 2025-12-04T09:48:06.8381026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8381533Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8381793Z 2025-12-04T09:48:06.8381897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8382244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8382551Z res = mod(**inputs) 2025-12-04T09:48:06.8382896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8383274Z outputs = self.roberta( 2025-12-04T09:48:06.8383635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8384016Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8384384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8384758Z layer_outputs = layer_module( 2025-12-04T09:48:06.8385097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8385439Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8385804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8386160Z return func(*args, **kwargs) 2025-12-04T09:48:06.8386528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8386911Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8387283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8387645Z return func(*args, **kwargs) 2025-12-04T09:48:06.8388005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8388386Z self_outputs = self.self( 2025-12-04T09:48:06.8388758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8389120Z return func(*args, **kwargs) 2025-12-04T09:48:06.8389490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8389880Z self.key(current_states) 2025-12-04T09:48:06.8390006Z 2025-12-04T09:48:06.8390118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8390476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8390802Z res = mod(**inputs) 2025-12-04T09:48:06.8391168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8391534Z outputs = self.roberta( 2025-12-04T09:48:06.8391898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8392268Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8392632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8393002Z layer_outputs = layer_module( 2025-12-04T09:48:06.8393326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8393669Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8394052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8394395Z return func(*args, **kwargs) 2025-12-04T09:48:06.8394756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8395137Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8395505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8395853Z return func(*args, **kwargs) 2025-12-04T09:48:06.8396219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8396592Z self_outputs = self.self( 2025-12-04T09:48:06.8396938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8397291Z return func(*args, **kwargs) 2025-12-04T09:48:06.8397656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8398030Z self.value(current_states) 2025-12-04T09:48:06.8398144Z 2025-12-04T09:48:06.8398226Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8398461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8398811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8399134Z res = mod(**inputs) 2025-12-04T09:48:06.8399511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8399927Z outputs = self.roberta( 2025-12-04T09:48:06.8400330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8400849Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8401296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8401737Z layer_outputs = layer_module( 2025-12-04T09:48:06.8402131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8402534Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8402966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8403318Z return func(*args, **kwargs) 2025-12-04T09:48:06.8403669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8404048Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8404410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8404781Z return func(*args, **kwargs) 2025-12-04T09:48:06.8405140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8405509Z self_outputs = self.self( 2025-12-04T09:48:06.8405875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8406236Z return func(*args, **kwargs) 2025-12-04T09:48:06.8406599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8407039Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8407218Z 2025-12-04T09:48:06.8407326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8407671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8408015Z res = mod(**inputs) 2025-12-04T09:48:06.8408369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8408742Z outputs = self.roberta( 2025-12-04T09:48:06.8409101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8409484Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8409860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8410230Z layer_outputs = layer_module( 2025-12-04T09:48:06.8410568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8410922Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8411296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8411646Z return func(*args, **kwargs) 2025-12-04T09:48:06.8412016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8412406Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8412780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8413133Z return func(*args, **kwargs) 2025-12-04T09:48:06.8413500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8413931Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8414354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8414749Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8414890Z 2025-12-04T09:48:06.8414992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8415339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8415646Z res = mod(**inputs) 2025-12-04T09:48:06.8416020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8416399Z outputs = self.roberta( 2025-12-04T09:48:06.8416763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8417135Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8417511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8417890Z layer_outputs = layer_module( 2025-12-04T09:48:06.8418239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8418597Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8418993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8419371Z return func(*args, **kwargs) 2025-12-04T09:48:06.8419730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8420120Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8420506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8420880Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8421300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8421796Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8422234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8422630Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8422776Z 2025-12-04T09:48:06.8422882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8423243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8423565Z res = mod(**inputs) 2025-12-04T09:48:06.8423925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8424311Z outputs = self.roberta( 2025-12-04T09:48:06.8424690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8425083Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8425472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8425862Z layer_outputs = layer_module( 2025-12-04T09:48:06.8426212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8426569Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8426970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8427340Z return func(*args, **kwargs) 2025-12-04T09:48:06.8427716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8428119Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8428518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8428915Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8429344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8429831Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8430294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8430737Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8431119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8431460Z return self.act(input) 2025-12-04T09:48:06.8431569Z 2025-12-04T09:48:06.8431680Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8432062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8432377Z res = mod(**inputs) 2025-12-04T09:48:06.8432745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8433153Z outputs = self.roberta( 2025-12-04T09:48:06.8433522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8433991Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8434375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8434760Z layer_outputs = layer_module( 2025-12-04T09:48:06.8435136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8435518Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8435897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8436261Z return func(*args, **kwargs) 2025-12-04T09:48:06.8436658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8437074Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8437483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8437879Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8438328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8438852Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8439343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8439776Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8439929Z 2025-12-04T09:48:06.8440044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8440518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8440873Z res = mod(**inputs) 2025-12-04T09:48:06.8441275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8441697Z outputs = self.roberta( 2025-12-04T09:48:06.8442108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8442527Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8442953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8443384Z layer_outputs = layer_module( 2025-12-04T09:48:06.8443760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8444166Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8444610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8445014Z return func(*args, **kwargs) 2025-12-04T09:48:06.8445419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8445887Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8446302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8446705Z return func(*args, **kwargs) 2025-12-04T09:48:06.8447260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8447692Z self_outputs = self.self( 2025-12-04T09:48:06.8448110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8448512Z return func(*args, **kwargs) 2025-12-04T09:48:06.8448935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8449514Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8449798Z 2025-12-04T09:48:06.8449924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8450311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8450756Z res = mod(**inputs) 2025-12-04T09:48:06.8451155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8451554Z outputs = self.roberta( 2025-12-04T09:48:06.8451921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8452325Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8452713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8453094Z layer_outputs = layer_module( 2025-12-04T09:48:06.8453444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8453803Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8454179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8454542Z return func(*args, **kwargs) 2025-12-04T09:48:06.8454918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8455314Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8455687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8456056Z return func(*args, **kwargs) 2025-12-04T09:48:06.8456433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8456820Z self_outputs = self.self( 2025-12-04T09:48:06.8457170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8457540Z return func(*args, **kwargs) 2025-12-04T09:48:06.8457923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8458315Z self.key(current_states) 2025-12-04T09:48:06.8458428Z 2025-12-04T09:48:06.8458532Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8458888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8459209Z res = mod(**inputs) 2025-12-04T09:48:06.8459595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8459981Z outputs = self.roberta( 2025-12-04T09:48:06.8460353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8460745Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8461122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8461541Z layer_outputs = layer_module( 2025-12-04T09:48:06.8461893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8462252Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8462664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8463033Z return func(*args, **kwargs) 2025-12-04T09:48:06.8463406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8463794Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8464176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8464555Z return func(*args, **kwargs) 2025-12-04T09:48:06.8464948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8465328Z self_outputs = self.self( 2025-12-04T09:48:06.8465681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8466046Z return func(*args, **kwargs) 2025-12-04T09:48:06.8466414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8466793Z self.value(current_states) 2025-12-04T09:48:06.8466916Z 2025-12-04T09:48:06.8466996Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8467230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8467569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8467879Z res = mod(**inputs) 2025-12-04T09:48:06.8468239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8468612Z outputs = self.roberta( 2025-12-04T09:48:06.8468981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8469364Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8469801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8470175Z layer_outputs = layer_module( 2025-12-04T09:48:06.8470514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8470871Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8471237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8471601Z return func(*args, **kwargs) 2025-12-04T09:48:06.8471973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8472360Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8472598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8472665Z return func(*args, **kwargs) 2025-12-04T09:48:06.8472944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8473015Z self_outputs = self.self( 2025-12-04T09:48:06.8473255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8473320Z return func(*args, **kwargs) 2025-12-04T09:48:06.8473572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8473727Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8473731Z 2025-12-04T09:48:06.8473835Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8474042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8474114Z res = mod(**inputs) 2025-12-04T09:48:06.8474365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8474438Z outputs = self.roberta( 2025-12-04T09:48:06.8474687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8474758Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8475016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8475130Z layer_outputs = layer_module( 2025-12-04T09:48:06.8475358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8475436Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8475678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8475754Z return func(*args, **kwargs) 2025-12-04T09:48:06.8476013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8476095Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8476344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8476412Z return func(*args, **kwargs) 2025-12-04T09:48:06.8476681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8476807Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8477067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8477160Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8477164Z 2025-12-04T09:48:06.8477272Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8477483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8477549Z res = mod(**inputs) 2025-12-04T09:48:06.8477813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8477891Z outputs = self.roberta( 2025-12-04T09:48:06.8478154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8478233Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8478508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8478580Z layer_outputs = layer_module( 2025-12-04T09:48:06.8478831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8478912Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8479156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8479233Z return func(*args, **kwargs) 2025-12-04T09:48:06.8479495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8479589Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8479893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8479977Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8480314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8480517Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8480813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8480917Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8480921Z 2025-12-04T09:48:06.8481037Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8481263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8481358Z res = mod(**inputs) 2025-12-04T09:48:06.8481653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8481732Z outputs = self.roberta( 2025-12-04T09:48:06.8482001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8482076Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8482351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8482426Z layer_outputs = layer_module( 2025-12-04T09:48:06.8482664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8482744Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8482994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8483075Z return func(*args, **kwargs) 2025-12-04T09:48:06.8483344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8483437Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8483700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8483781Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8484089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8484211Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8484478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8484602Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8484828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8484907Z return self.act(input) 2025-12-04T09:48:06.8484910Z 2025-12-04T09:48:06.8485015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8485216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8485307Z res = mod(**inputs) 2025-12-04T09:48:06.8485572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8485649Z outputs = self.roberta( 2025-12-04T09:48:06.8485910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8485984Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8486255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8486346Z layer_outputs = layer_module( 2025-12-04T09:48:06.8486576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8486688Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8486940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8487015Z return func(*args, **kwargs) 2025-12-04T09:48:06.8487291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8487375Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8487652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8487784Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8488088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8488224Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8488488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8488579Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8488583Z 2025-12-04T09:48:06.8488688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8488886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8488960Z res = mod(**inputs) 2025-12-04T09:48:06.8489224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8489303Z outputs = self.roberta( 2025-12-04T09:48:06.8489568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8489641Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8489915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8489983Z layer_outputs = layer_module( 2025-12-04T09:48:06.8490206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8490281Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8490513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8490586Z return func(*args, **kwargs) 2025-12-04T09:48:06.8490836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8490920Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8491159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8491225Z return func(*args, **kwargs) 2025-12-04T09:48:06.8491482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8491580Z self_outputs = self.self( 2025-12-04T09:48:06.8491813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8491888Z return func(*args, **kwargs) 2025-12-04T09:48:06.8492135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8492335Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8492376Z 2025-12-04T09:48:06.8492478Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8492669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8492739Z res = mod(**inputs) 2025-12-04T09:48:06.8493007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8493076Z outputs = self.roberta( 2025-12-04T09:48:06.8493339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8493411Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8493671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8493740Z layer_outputs = layer_module( 2025-12-04T09:48:06.8493972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8494057Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8494288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8494355Z return func(*args, **kwargs) 2025-12-04T09:48:06.8494611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8494690Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8494925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8494991Z return func(*args, **kwargs) 2025-12-04T09:48:06.8495238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8495316Z self_outputs = self.self( 2025-12-04T09:48:06.8495544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8495615Z return func(*args, **kwargs) 2025-12-04T09:48:06.8495864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8495932Z self.key(current_states) 2025-12-04T09:48:06.8495938Z 2025-12-04T09:48:06.8496044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8496233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8496295Z res = mod(**inputs) 2025-12-04T09:48:06.8496551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8496616Z outputs = self.roberta( 2025-12-04T09:48:06.8496876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8496948Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8497197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8497273Z layer_outputs = layer_module( 2025-12-04T09:48:06.8497507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8497585Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8497825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8497892Z return func(*args, **kwargs) 2025-12-04T09:48:06.8498147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8498246Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8498481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8498553Z return func(*args, **kwargs) 2025-12-04T09:48:06.8498821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8498902Z self_outputs = self.self( 2025-12-04T09:48:06.8499139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8499207Z return func(*args, **kwargs) 2025-12-04T09:48:06.8499469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8499541Z self.value(current_states) 2025-12-04T09:48:06.8499544Z 2025-12-04T09:48:06.8499642Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8499753Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8499947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8500018Z res = mod(**inputs) 2025-12-04T09:48:06.8500273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8500340Z outputs = self.roberta( 2025-12-04T09:48:06.8500601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8500672Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8500932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8501007Z layer_outputs = layer_module( 2025-12-04T09:48:06.8501222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8501305Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8501535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8501602Z return func(*args, **kwargs) 2025-12-04T09:48:06.8501859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8501938Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8502172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8502238Z return func(*args, **kwargs) 2025-12-04T09:48:06.8502486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8502562Z self_outputs = self.self( 2025-12-04T09:48:06.8502793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8502857Z return func(*args, **kwargs) 2025-12-04T09:48:06.8503115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8503243Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8503263Z 2025-12-04T09:48:06.8503372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8503561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8503623Z res = mod(**inputs) 2025-12-04T09:48:06.8503878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8503944Z outputs = self.roberta( 2025-12-04T09:48:06.8504195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8504291Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8504542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8504633Z layer_outputs = layer_module( 2025-12-04T09:48:06.8504854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8504931Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8505171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8505237Z return func(*args, **kwargs) 2025-12-04T09:48:06.8505504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8505598Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8505823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8505893Z return func(*args, **kwargs) 2025-12-04T09:48:06.8506144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8506266Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8506518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8506595Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8506599Z 2025-12-04T09:48:06.8506704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8506889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8506951Z res = mod(**inputs) 2025-12-04T09:48:06.8507199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8507263Z outputs = self.roberta( 2025-12-04T09:48:06.8507517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8507589Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8507840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8507914Z layer_outputs = layer_module( 2025-12-04T09:48:06.8508126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8508201Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8508439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8508514Z return func(*args, **kwargs) 2025-12-04T09:48:06.8508765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8508848Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8509110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8509195Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8509477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8509600Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8509851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8509933Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8509952Z 2025-12-04T09:48:06.8510059Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8510250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8510313Z res = mod(**inputs) 2025-12-04T09:48:06.8510585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8510653Z outputs = self.roberta( 2025-12-04T09:48:06.8510925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8510992Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8511237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8511313Z layer_outputs = layer_module( 2025-12-04T09:48:06.8511538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8511614Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8511845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8511912Z return func(*args, **kwargs) 2025-12-04T09:48:06.8512161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8512240Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8512481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8512560Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8512831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8512952Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8513194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8513298Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8513509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8513575Z return self.act(input) 2025-12-04T09:48:06.8513578Z 2025-12-04T09:48:06.8513682Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8513866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8513928Z res = mod(**inputs) 2025-12-04T09:48:06.8514178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8514244Z outputs = self.roberta( 2025-12-04T09:48:06.8514486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8514560Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8514802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8514900Z layer_outputs = layer_module( 2025-12-04T09:48:06.8515113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8515187Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8515421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8515485Z return func(*args, **kwargs) 2025-12-04T09:48:06.8515732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8515835Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8516075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8516172Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8516451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8516577Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8516827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8516904Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8516907Z 2025-12-04T09:48:06.8517009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8517208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8517271Z res = mod(**inputs) 2025-12-04T09:48:06.8517520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8517585Z outputs = self.roberta( 2025-12-04T09:48:06.8517830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8517905Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8518147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8518219Z layer_outputs = layer_module( 2025-12-04T09:48:06.8518431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8518507Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8518747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8518814Z return func(*args, **kwargs) 2025-12-04T09:48:06.8519069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8519148Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8519381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8519455Z return func(*args, **kwargs) 2025-12-04T09:48:06.8519719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8519792Z self_outputs = self.self( 2025-12-04T09:48:06.8520043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8520115Z return func(*args, **kwargs) 2025-12-04T09:48:06.8520386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8520687Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8520693Z 2025-12-04T09:48:06.8520807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8521054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8521129Z res = mod(**inputs) 2025-12-04T09:48:06.8521424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8521495Z outputs = self.roberta( 2025-12-04T09:48:06.8521765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8521872Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8522143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8522217Z layer_outputs = layer_module( 2025-12-04T09:48:06.8522478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8522562Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8522817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8522888Z return func(*args, **kwargs) 2025-12-04T09:48:06.8523156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8523246Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8523511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8523582Z return func(*args, **kwargs) 2025-12-04T09:48:06.8523853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8523926Z self_outputs = self.self( 2025-12-04T09:48:06.8524177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8524248Z return func(*args, **kwargs) 2025-12-04T09:48:06.8524509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8524588Z self.key(current_states) 2025-12-04T09:48:06.8524591Z 2025-12-04T09:48:06.8524694Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8524900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8524971Z res = mod(**inputs) 2025-12-04T09:48:06.8525233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8525310Z outputs = self.roberta( 2025-12-04T09:48:06.8525570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8525645Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8525912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8525983Z layer_outputs = layer_module( 2025-12-04T09:48:06.8526215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8526295Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8526542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8526620Z return func(*args, **kwargs) 2025-12-04T09:48:06.8526883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8526967Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8527235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8527306Z return func(*args, **kwargs) 2025-12-04T09:48:06.8527580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8527651Z self_outputs = self.self( 2025-12-04T09:48:06.8527897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8527976Z return func(*args, **kwargs) 2025-12-04T09:48:06.8528253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8528332Z self.value(current_states) 2025-12-04T09:48:06.8528336Z 2025-12-04T09:48:06.8528420Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8528537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8528748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8528814Z res = mod(**inputs) 2025-12-04T09:48:06.8529074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8529152Z outputs = self.roberta( 2025-12-04T09:48:06.8529413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8529519Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8529782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8529853Z layer_outputs = layer_module( 2025-12-04T09:48:06.8530086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8530168Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8530414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8530491Z return func(*args, **kwargs) 2025-12-04T09:48:06.8530752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8530842Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8531086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8531158Z return func(*args, **kwargs) 2025-12-04T09:48:06.8531428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8531498Z self_outputs = self.self( 2025-12-04T09:48:06.8531752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8531821Z return func(*args, **kwargs) 2025-12-04T09:48:06.8532082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8532222Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8532225Z 2025-12-04T09:48:06.8532329Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8532528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8532605Z res = mod(**inputs) 2025-12-04T09:48:06.8532867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8532941Z outputs = self.roberta( 2025-12-04T09:48:06.8533204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8533296Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8533577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8533645Z layer_outputs = layer_module( 2025-12-04T09:48:06.8533870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8533946Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8534184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8534293Z return func(*args, **kwargs) 2025-12-04T09:48:06.8534545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8534642Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8534902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8534970Z return func(*args, **kwargs) 2025-12-04T09:48:06.8535230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8535355Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8535609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8535725Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8535728Z 2025-12-04T09:48:06.8535829Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8536030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8536095Z res = mod(**inputs) 2025-12-04T09:48:06.8536352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8536426Z outputs = self.roberta( 2025-12-04T09:48:06.8536683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8536754Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8537021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8537092Z layer_outputs = layer_module( 2025-12-04T09:48:06.8537318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8537392Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8537630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8537701Z return func(*args, **kwargs) 2025-12-04T09:48:06.8537961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8538042Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8538300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8538375Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8538672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8538790Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8539047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8539134Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8539138Z 2025-12-04T09:48:06.8539236Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8539454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8539519Z res = mod(**inputs) 2025-12-04T09:48:06.8539771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8539844Z outputs = self.roberta( 2025-12-04T09:48:06.8540096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8540184Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8540445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8540514Z layer_outputs = layer_module( 2025-12-04T09:48:06.8540754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8540833Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8541066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8541140Z return func(*args, **kwargs) 2025-12-04T09:48:06.8541390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8541479Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8541742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8541820Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8542113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8542228Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8542478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8542594Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8542801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8542875Z return self.act(input) 2025-12-04T09:48:06.8542878Z 2025-12-04T09:48:06.8542975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8543167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8543239Z res = mod(**inputs) 2025-12-04T09:48:06.8543486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8543559Z outputs = self.roberta( 2025-12-04T09:48:06.8543808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8543878Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8544134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8544200Z layer_outputs = layer_module( 2025-12-04T09:48:06.8544413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8544498Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8544730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8544802Z return func(*args, **kwargs) 2025-12-04T09:48:06.8545056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8545136Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8545407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8545484Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8545765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8545900Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8546154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8546260Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8546263Z 2025-12-04T09:48:06.8546362Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8546571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8546642Z res = mod(**inputs) 2025-12-04T09:48:06.8546891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8546966Z outputs = self.roberta( 2025-12-04T09:48:06.8547371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8547446Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8547704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8547814Z layer_outputs = layer_module( 2025-12-04T09:48:06.8548029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8548114Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8548351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8548425Z return func(*args, **kwargs) 2025-12-04T09:48:06.8548674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8548754Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8548994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8549061Z return func(*args, **kwargs) 2025-12-04T09:48:06.8549319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8549389Z self_outputs = self.self( 2025-12-04T09:48:06.8549623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8549698Z return func(*args, **kwargs) 2025-12-04T09:48:06.8549951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8550153Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8550165Z 2025-12-04T09:48:06.8550268Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8550458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8550532Z res = mod(**inputs) 2025-12-04T09:48:06.8550783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8550850Z outputs = self.roberta( 2025-12-04T09:48:06.8551112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8551185Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8551463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8551533Z layer_outputs = layer_module( 2025-12-04T09:48:06.8551749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8551832Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8552068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8552156Z return func(*args, **kwargs) 2025-12-04T09:48:06.8552422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8552502Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8552781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8552851Z return func(*args, **kwargs) 2025-12-04T09:48:06.8553100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8553177Z self_outputs = self.self( 2025-12-04T09:48:06.8553414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8553479Z return func(*args, **kwargs) 2025-12-04T09:48:06.8553754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8553836Z self.key(current_states) 2025-12-04T09:48:06.8553839Z 2025-12-04T09:48:06.8553943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8554130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8554191Z res = mod(**inputs) 2025-12-04T09:48:06.8554442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8554505Z outputs = self.roberta( 2025-12-04T09:48:06.8554755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8554822Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8555066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8555143Z layer_outputs = layer_module( 2025-12-04T09:48:06.8555354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8555428Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8555668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8555734Z return func(*args, **kwargs) 2025-12-04T09:48:06.8555985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8556064Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8556291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8556364Z return func(*args, **kwargs) 2025-12-04T09:48:06.8556610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8556685Z self_outputs = self.self( 2025-12-04T09:48:06.8556916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8556982Z return func(*args, **kwargs) 2025-12-04T09:48:06.8559679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8559762Z self.value(current_states) 2025-12-04T09:48:06.8559765Z 2025-12-04T09:48:06.8559847Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8559959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8560151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8560215Z res = mod(**inputs) 2025-12-04T09:48:06.8560540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8560612Z outputs = self.roberta( 2025-12-04T09:48:06.8560889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8561019Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8561347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8561436Z layer_outputs = layer_module( 2025-12-04T09:48:06.8561684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8561773Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8562038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8562132Z return func(*args, **kwargs) 2025-12-04T09:48:06.8562407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8562498Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8562740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8562825Z return func(*args, **kwargs) 2025-12-04T09:48:06.8563083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8563151Z self_outputs = self.self( 2025-12-04T09:48:06.8563393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8563458Z return func(*args, **kwargs) 2025-12-04T09:48:06.8563712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8563850Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8563854Z 2025-12-04T09:48:06.8563955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8564156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8564221Z res = mod(**inputs) 2025-12-04T09:48:06.8564484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8564557Z outputs = self.roberta( 2025-12-04T09:48:06.8564819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8564902Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8565164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8565235Z layer_outputs = layer_module( 2025-12-04T09:48:06.8565467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8565546Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8565790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8565935Z return func(*args, **kwargs) 2025-12-04T09:48:06.8566208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8566299Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8566537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8566604Z return func(*args, **kwargs) 2025-12-04T09:48:06.8566866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8566997Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8567261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8567357Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8567362Z 2025-12-04T09:48:06.8567468Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8567680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8567743Z res = mod(**inputs) 2025-12-04T09:48:06.8567994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8568067Z outputs = self.roberta( 2025-12-04T09:48:06.8568323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8568419Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8568675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8568745Z layer_outputs = layer_module( 2025-12-04T09:48:06.8568977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8569059Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8569298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8569374Z return func(*args, **kwargs) 2025-12-04T09:48:06.8569631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8569722Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8569978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8570054Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8570352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8570472Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8570739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8570822Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8570825Z 2025-12-04T09:48:06.8570933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8571135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8571200Z res = mod(**inputs) 2025-12-04T09:48:06.8571453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8571527Z outputs = self.roberta( 2025-12-04T09:48:06.8571787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8571865Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8572173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8572244Z layer_outputs = layer_module( 2025-12-04T09:48:06.8572471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8572547Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8572790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8572861Z return func(*args, **kwargs) 2025-12-04T09:48:06.8573117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8573205Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8573473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8573554Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8573851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8573968Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8574231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8574359Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8574574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8574652Z return self.act(input) 2025-12-04T09:48:06.8574655Z 2025-12-04T09:48:06.8574757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8574960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8575029Z res = mod(**inputs) 2025-12-04T09:48:06.8575287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8575363Z outputs = self.roberta( 2025-12-04T09:48:06.8575621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8575694Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8575960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8576032Z layer_outputs = layer_module( 2025-12-04T09:48:06.8576261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8576339Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8576580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8576657Z return func(*args, **kwargs) 2025-12-04T09:48:06.8576915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8576996Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8577258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8577335Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8577633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8577764Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8578025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8578149Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8578152Z 2025-12-04T09:48:06.8578255Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8578456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8578521Z res = mod(**inputs) 2025-12-04T09:48:06.8578776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8578852Z outputs = self.roberta( 2025-12-04T09:48:06.8579109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8579186Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8579454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8579527Z layer_outputs = layer_module( 2025-12-04T09:48:06.8579755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8579832Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8580071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8580146Z return func(*args, **kwargs) 2025-12-04T09:48:06.8580402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8580504Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8580749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8580815Z return func(*args, **kwargs) 2025-12-04T09:48:06.8581072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8581142Z self_outputs = self.self( 2025-12-04T09:48:06.8581374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8581447Z return func(*args, **kwargs) 2025-12-04T09:48:06.8581694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8581904Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8581909Z 2025-12-04T09:48:06.8582007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8582195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8582266Z res = mod(**inputs) 2025-12-04T09:48:06.8582518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8582595Z outputs = self.roberta( 2025-12-04T09:48:06.8582847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8582917Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8583173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8583242Z layer_outputs = layer_module( 2025-12-04T09:48:06.8583458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8583542Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8583775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8583851Z return func(*args, **kwargs) 2025-12-04T09:48:06.8584150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8584232Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8584472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8584538Z return func(*args, **kwargs) 2025-12-04T09:48:06.8584795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8584863Z self_outputs = self.self( 2025-12-04T09:48:06.8585092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8585163Z return func(*args, **kwargs) 2025-12-04T09:48:06.8585453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8585525Z self.key(current_states) 2025-12-04T09:48:06.8585530Z 2025-12-04T09:48:06.8585639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8585827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8585894Z res = mod(**inputs) 2025-12-04T09:48:06.8586153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8586220Z outputs = self.roberta( 2025-12-04T09:48:06.8586492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8586563Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8586819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8586898Z layer_outputs = layer_module( 2025-12-04T09:48:06.8587119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8587203Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8587440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8587506Z return func(*args, **kwargs) 2025-12-04T09:48:06.8587769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8587851Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8588099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8588165Z return func(*args, **kwargs) 2025-12-04T09:48:06.8588415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8588492Z self_outputs = self.self( 2025-12-04T09:48:06.8588725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8588790Z return func(*args, **kwargs) 2025-12-04T09:48:06.8589049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8589121Z self.value(current_states) 2025-12-04T09:48:06.8589124Z 2025-12-04T09:48:06.8589215Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8589317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8589509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8589590Z res = mod(**inputs) 2025-12-04T09:48:06.8589839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8589923Z outputs = self.roberta( 2025-12-04T09:48:06.8590201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8590273Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8590535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8590602Z layer_outputs = layer_module( 2025-12-04T09:48:06.8590819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8590904Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8591142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8591215Z return func(*args, **kwargs) 2025-12-04T09:48:06.8591485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8591567Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8591806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8591871Z return func(*args, **kwargs) 2025-12-04T09:48:06.8592119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8592209Z self_outputs = self.self( 2025-12-04T09:48:06.8592443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8592513Z return func(*args, **kwargs) 2025-12-04T09:48:06.8592765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8592892Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8592896Z 2025-12-04T09:48:06.8593002Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8593193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8593255Z res = mod(**inputs) 2025-12-04T09:48:06.8593516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8593583Z outputs = self.roberta( 2025-12-04T09:48:06.8593845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8593913Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8594166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8594243Z layer_outputs = layer_module( 2025-12-04T09:48:06.8594462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8594545Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8594781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8594847Z return func(*args, **kwargs) 2025-12-04T09:48:06.8595103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8595183Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8595416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8595487Z return func(*args, **kwargs) 2025-12-04T09:48:06.8595742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8595914Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8596170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8596253Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8596257Z 2025-12-04T09:48:06.8596366Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8596562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8596636Z res = mod(**inputs) 2025-12-04T09:48:06.8596895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8596964Z outputs = self.roberta( 2025-12-04T09:48:06.8597250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8597326Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8597595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8597674Z layer_outputs = layer_module( 2025-12-04T09:48:06.8597902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8597989Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8598237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8598328Z return func(*args, **kwargs) 2025-12-04T09:48:06.8598598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8598685Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8598960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8599040Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8599341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8599479Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8599758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8599850Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8599854Z 2025-12-04T09:48:06.8599972Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8600185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8600260Z res = mod(**inputs) 2025-12-04T09:48:06.8600631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8600712Z outputs = self.roberta( 2025-12-04T09:48:06.8601004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8601086Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8601376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8601452Z layer_outputs = layer_module( 2025-12-04T09:48:06.8601693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8601787Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8602049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8602131Z return func(*args, **kwargs) 2025-12-04T09:48:06.8602440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8602525Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8602787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8602864Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8603152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8603280Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8603535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8603646Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8603882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8603958Z return self.act(input) 2025-12-04T09:48:06.8603962Z 2025-12-04T09:48:06.8604071Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8604268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8604331Z res = mod(**inputs) 2025-12-04T09:48:06.8604595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8604679Z outputs = self.roberta( 2025-12-04T09:48:06.8604940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8605012Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8605270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8605349Z layer_outputs = layer_module( 2025-12-04T09:48:06.8605571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8605652Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8605900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8605967Z return func(*args, **kwargs) 2025-12-04T09:48:06.8606229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8606312Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8606564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8606648Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8606935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8607073Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8607332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8607412Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8607415Z 2025-12-04T09:48:06.8607524Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8607722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8607786Z res = mod(**inputs) 2025-12-04T09:48:06.8608048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8608116Z outputs = self.roberta( 2025-12-04T09:48:06.8608404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8608492Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8608753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8608831Z layer_outputs = layer_module( 2025-12-04T09:48:06.8609054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8609137Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8609384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8609453Z return func(*args, **kwargs) 2025-12-04T09:48:06.8609737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8609821Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8610064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8610140Z return func(*args, **kwargs) 2025-12-04T09:48:06.8610397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8610473Z self_outputs = self.self( 2025-12-04T09:48:06.8610712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8610798Z return func(*args, **kwargs) 2025-12-04T09:48:06.8611062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8611269Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8611273Z 2025-12-04T09:48:06.8611384Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8611580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8611646Z res = mod(**inputs) 2025-12-04T09:48:06.8611917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8611984Z outputs = self.roberta( 2025-12-04T09:48:06.8612243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8612324Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8612582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8612657Z layer_outputs = layer_module( 2025-12-04T09:48:06.8612881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8612965Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8613213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8613281Z return func(*args, **kwargs) 2025-12-04T09:48:06.8613540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8613631Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8613873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8613950Z return func(*args, **kwargs) 2025-12-04T09:48:06.8614210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8614280Z self_outputs = self.self( 2025-12-04T09:48:06.8614560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8614628Z return func(*args, **kwargs) 2025-12-04T09:48:06.8614892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8614962Z self.key(current_states) 2025-12-04T09:48:06.8614965Z 2025-12-04T09:48:06.8615067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8615266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8615333Z res = mod(**inputs) 2025-12-04T09:48:06.8615591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8615666Z outputs = self.roberta( 2025-12-04T09:48:06.8615949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8616032Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8616281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8616348Z layer_outputs = layer_module( 2025-12-04T09:48:06.8616568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8616644Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8616893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8616966Z return func(*args, **kwargs) 2025-12-04T09:48:06.8617215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8617303Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8617537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8617602Z return func(*args, **kwargs) 2025-12-04T09:48:06.8617857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8617925Z self_outputs = self.self( 2025-12-04T09:48:06.8618165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8618232Z return func(*args, **kwargs) 2025-12-04T09:48:06.8618481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8618558Z self.value(current_states) 2025-12-04T09:48:06.8618562Z 2025-12-04T09:48:06.8618640Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8618742Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8618941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8619004Z res = mod(**inputs) 2025-12-04T09:48:06.8619261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8619327Z outputs = self.roberta( 2025-12-04T09:48:06.8619580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8619659Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8619908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8619976Z layer_outputs = layer_module( 2025-12-04T09:48:06.8620196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8620294Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8620549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8620617Z return func(*args, **kwargs) 2025-12-04T09:48:06.8620865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8620951Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8621183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8621259Z return func(*args, **kwargs) 2025-12-04T09:48:06.8621507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8621573Z self_outputs = self.self( 2025-12-04T09:48:06.8621822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8621891Z return func(*args, **kwargs) 2025-12-04T09:48:06.8622142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8622282Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8622285Z 2025-12-04T09:48:06.8622386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8622587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8622669Z res = mod(**inputs) 2025-12-04T09:48:06.8622943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8623016Z outputs = self.roberta( 2025-12-04T09:48:06.8623270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8623351Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8623605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8623672Z layer_outputs = layer_module( 2025-12-04T09:48:06.8623892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8623967Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8624202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8624274Z return func(*args, **kwargs) 2025-12-04T09:48:06.8624530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8624619Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8624856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8624921Z return func(*args, **kwargs) 2025-12-04T09:48:06.8625181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8625309Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8625567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8625658Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8625661Z 2025-12-04T09:48:06.8625762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8625963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8626029Z res = mod(**inputs) 2025-12-04T09:48:06.8626308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8626400Z outputs = self.roberta( 2025-12-04T09:48:06.8626665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8626745Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8626998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8627068Z layer_outputs = layer_module( 2025-12-04T09:48:06.8627291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8627370Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8627620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8627696Z return func(*args, **kwargs) 2025-12-04T09:48:06.8627950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8628041Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8628288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8628361Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8628650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8628792Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8629050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8629129Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8629132Z 2025-12-04T09:48:06.8629233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8629433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8629498Z res = mod(**inputs) 2025-12-04T09:48:06.8629748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8629821Z outputs = self.roberta( 2025-12-04T09:48:06.8630074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8630153Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8630405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8630472Z layer_outputs = layer_module( 2025-12-04T09:48:06.8630701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8630782Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8631036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8631104Z return func(*args, **kwargs) 2025-12-04T09:48:06.8631362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8631452Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8631717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8631793Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8632090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8632210Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8632505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8632617Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8632832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8632911Z return self.act(input) 2025-12-04T09:48:06.8632915Z 2025-12-04T09:48:06.8633017Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8633222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8633286Z res = mod(**inputs) 2025-12-04T09:48:06.8633544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8633631Z outputs = self.roberta( 2025-12-04T09:48:06.8633890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8633964Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8634225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8634295Z layer_outputs = layer_module( 2025-12-04T09:48:06.8634520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8634614Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8634854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8634928Z return func(*args, **kwargs) 2025-12-04T09:48:06.8635183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8635267Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8635533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8635608Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8635904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8636034Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8636294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8636383Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8636387Z 2025-12-04T09:48:06.8636486Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8636687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8636755Z res = mod(**inputs) 2025-12-04T09:48:06.8637008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8637084Z outputs = self.roberta( 2025-12-04T09:48:06.8637338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8637409Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8637668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8637739Z layer_outputs = layer_module( 2025-12-04T09:48:06.8637963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8638040Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8638279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8638386Z return func(*args, **kwargs) 2025-12-04T09:48:06.8638645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8638732Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8638969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8639037Z return func(*args, **kwargs) 2025-12-04T09:48:06.8639302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8639371Z self_outputs = self.self( 2025-12-04T09:48:06.8639644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8639736Z return func(*args, **kwargs) 2025-12-04T09:48:06.8640004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:48:06.8640223Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:48:06.8640226Z 2025-12-04T09:48:06.8640331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8640734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8640815Z res = mod(**inputs) 2025-12-04T09:48:06.8641102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8641180Z outputs = self.roberta( 2025-12-04T09:48:06.8641470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8641550Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8641840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8641917Z layer_outputs = layer_module( 2025-12-04T09:48:06.8642158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8642253Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8642516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8642600Z return func(*args, **kwargs) 2025-12-04T09:48:06.8642908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8642991Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8643240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8643313Z return func(*args, **kwargs) 2025-12-04T09:48:06.8643570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8643648Z self_outputs = self.self( 2025-12-04T09:48:06.8643889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8643964Z return func(*args, **kwargs) 2025-12-04T09:48:06.8644221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:48:06.8644293Z self.key(current_states) 2025-12-04T09:48:06.8644297Z 2025-12-04T09:48:06.8644408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8644648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8644718Z res = mod(**inputs) 2025-12-04T09:48:06.8645015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8645084Z outputs = self.roberta( 2025-12-04T09:48:06.8645343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8645414Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8645669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8645750Z layer_outputs = layer_module( 2025-12-04T09:48:06.8645969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8646054Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8646308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8646380Z return func(*args, **kwargs) 2025-12-04T09:48:06.8646644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8646726Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8646974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8647297Z return func(*args, **kwargs) 2025-12-04T09:48:06.8647610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8647689Z self_outputs = self.self( 2025-12-04T09:48:06.8647931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8648001Z return func(*args, **kwargs) 2025-12-04T09:48:06.8648268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:48:06.8648341Z self.value(current_states) 2025-12-04T09:48:06.8648344Z 2025-12-04T09:48:06.8648436Z cudagraph partition due to non gpu ops 2025-12-04T09:48:06.8648540Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8648775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8648847Z res = mod(**inputs) 2025-12-04T09:48:06.8649102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8649170Z outputs = self.roberta( 2025-12-04T09:48:06.8649441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8649512Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8649771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8649843Z layer_outputs = layer_module( 2025-12-04T09:48:06.8650062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8650149Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8650388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8650456Z return func(*args, **kwargs) 2025-12-04T09:48:06.8650731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8650811Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8651054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8651153Z return func(*args, **kwargs) 2025-12-04T09:48:06.8651430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:48:06.8651504Z self_outputs = self.self( 2025-12-04T09:48:06.8651743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8651816Z return func(*args, **kwargs) 2025-12-04T09:48:06.8652076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:48:06.8652204Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:48:06.8652208Z 2025-12-04T09:48:06.8652315Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8652508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8652600Z res = mod(**inputs) 2025-12-04T09:48:06.8652861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8652927Z outputs = self.roberta( 2025-12-04T09:48:06.8653179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8653249Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8653498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8653590Z layer_outputs = layer_module( 2025-12-04T09:48:06.8653807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8653888Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8654128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8654194Z return func(*args, **kwargs) 2025-12-04T09:48:06.8654457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:48:06.8654535Z self_attention_outputs = self.attention( 2025-12-04T09:48:06.8654774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8654848Z return func(*args, **kwargs) 2025-12-04T09:48:06.8655102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:48:06.8655234Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:06.8655489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:48:06.8655570Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8655574Z 2025-12-04T09:48:06.8655682Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8655872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8655940Z res = mod(**inputs) 2025-12-04T09:48:06.8656195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8656263Z outputs = self.roberta( 2025-12-04T09:48:06.8656527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8656599Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8656854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8656930Z layer_outputs = layer_module( 2025-12-04T09:48:06.8657150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8657270Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8657500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8657566Z return func(*args, **kwargs) 2025-12-04T09:48:06.8657822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8657905Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8658157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8658238Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8658535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8658659Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8658912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:48:06.8658993Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8658996Z 2025-12-04T09:48:06.8659102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8659297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8659367Z res = mod(**inputs) 2025-12-04T09:48:06.8659632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8659699Z outputs = self.roberta( 2025-12-04T09:48:06.8659953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8660026Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8660275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8660349Z layer_outputs = layer_module( 2025-12-04T09:48:06.8660562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8660646Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8660875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8660942Z return func(*args, **kwargs) 2025-12-04T09:48:06.8661202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8661282Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8661533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8661610Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8661892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:48:06.8662015Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:06.8662269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:48:06.8662378Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:06.8662606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:06.8662674Z return self.act(input) 2025-12-04T09:48:06.8662677Z 2025-12-04T09:48:06.8662782Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8662973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8663080Z res = mod(**inputs) 2025-12-04T09:48:06.8663355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:48:06.8663422Z outputs = self.roberta( 2025-12-04T09:48:06.8663680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:48:06.8663751Z encoder_outputs = self.encoder( 2025-12-04T09:48:06.8664002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:48:06.8664077Z layer_outputs = layer_module( 2025-12-04T09:48:06.8664292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:06.8664368Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:06.8664625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:06.8664693Z return func(*args, **kwargs) 2025-12-04T09:48:06.8664950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:48:06.8665034Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:06.8665280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:06.8665381Z return forward_fn(*input_tensors) 2025-12-04T09:48:06.8665672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:48:06.8665810Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:06.8666070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:48:06.8666156Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:06.8666159Z 2025-12-04T09:48:06.8666270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8666466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8666530Z res = mod(**inputs) 2025-12-04T09:48:06.8666796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 988, in forward 2025-12-04T09:48:06.8666898Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T09:48:06.8667169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1121, in forward 2025-12-04T09:48:06.8667251Z x = self.dense(features) 2025-12-04T09:48:06.8667254Z 2025-12-04T09:48:06.8667355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8667552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8667617Z res = mod(**inputs) 2025-12-04T09:48:06.8667875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 988, in forward 2025-12-04T09:48:06.8667969Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T09:48:06.8668226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1126, in forward 2025-12-04T09:48:06.8668296Z x = self.decoder(x) 2025-12-04T09:48:06.8668301Z 2025-12-04T09:48:06.8668399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:06.8668591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:06.8668661Z res = mod(**inputs) 2025-12-04T09:48:06.8668920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 994, in forward 2025-12-04T09:48:06.8669021Z lm_loss = self.loss_function( 2025-12-04T09:48:06.8669283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:48:06.8669457Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:48:06.8669717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:48:06.8669910Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:48:06.8669915Z 2025-12-04T09:48:16.9993337Z Compilation time (from dynamo_timed): 16.685071549 2025-12-04T09:48:17.0121805Z pass 2025-12-04T09:48:17.0122358Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:17.0130503Z TIMING: _recursive_pre_grad_passes:0.00646 _recursive_joint_graph_passes:0.66058 _recursive_post_grad_passes:0.06746 async_compile.wait:0.77487 code_gen:9.56632 inductor_compile:10.8732 backend_compile:13.90885 gc:0.00138 entire_frame_compile:16.68507 total_wall_time:16.68507 2025-12-04T09:48:17.0135905Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:7229 | FakeTensor.__torch_dispatch__:4402 | ProxyTorchDispatchMode.__torch_dispatch__:1992 2025-12-04T09:48:17.0136486Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-12-04T09:48:19.4834250Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:48:19.4836308Z import pynvml # type: ignore[import] 2025-12-04T09:48:22.9243420Z 2025-12-04T09:48:23.7958595Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:48:23.7958903Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:48:23.7978515Z cpu eval T5ForConditionalGeneration 2025-12-04T09:48:24.9684113Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:25.3620714Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:25.7702993Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:34.3510218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3510816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3511222Z res = mod(**inputs) 2025-12-04T09:48:34.3511636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3512058Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3512486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3512921Z layer_outputs = layer_module( 2025-12-04T09:48:34.3513338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3513780Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3514210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3514627Z return func(*args, **kwargs) 2025-12-04T09:48:34.3515025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3515437Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3515857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3516272Z return func(*args, **kwargs) 2025-12-04T09:48:34.3517019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3517439Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3517859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3518249Z return func(*args, **kwargs) 2025-12-04T09:48:34.3518643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 549, in forward 2025-12-04T09:48:34.3519081Z position_bias = position_bias + causal_mask 2025-12-04T09:48:34.3519248Z 2025-12-04T09:48:34.3519374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3519787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3520209Z res = mod(**inputs) 2025-12-04T09:48:34.3520914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3521381Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3521809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3522231Z layer_outputs = layer_module( 2025-12-04T09:48:34.3522663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3523074Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3523613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3524030Z return func(*args, **kwargs) 2025-12-04T09:48:34.3524442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3524853Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3525294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3525703Z return func(*args, **kwargs) 2025-12-04T09:48:34.3526117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3526545Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3526965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3527419Z return func(*args, **kwargs) 2025-12-04T09:48:34.3527815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3528238Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3528390Z 2025-12-04T09:48:34.3528529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3528934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3529293Z res = mod(**inputs) 2025-12-04T09:48:34.3530562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3531240Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3531792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3532402Z layer_outputs = layer_module( 2025-12-04T09:48:34.3532988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3533729Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3534368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3534820Z return func(*args, **kwargs) 2025-12-04T09:48:34.3535588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3536011Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3536441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3536857Z return func(*args, **kwargs) 2025-12-04T09:48:34.3537255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3537679Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3538111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3538535Z return func(*args, **kwargs) 2025-12-04T09:48:34.3538984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3539604Z key_states = self.k(current_states) 2025-12-04T09:48:34.3539767Z 2025-12-04T09:48:34.3539902Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3540319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3540676Z res = mod(**inputs) 2025-12-04T09:48:34.3541085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3541515Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3542092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3542573Z layer_outputs = layer_module( 2025-12-04T09:48:34.3542984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3543405Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3543838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3544260Z return func(*args, **kwargs) 2025-12-04T09:48:34.3544673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3545119Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3545536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3546161Z return func(*args, **kwargs) 2025-12-04T09:48:34.3546563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3546996Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3547573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3547976Z return func(*args, **kwargs) 2025-12-04T09:48:34.3548371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3548774Z value_states = self.v(current_states) 2025-12-04T09:48:34.3548936Z 2025-12-04T09:48:34.3549028Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3549265Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3549529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3549921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3550412Z res = mod(**inputs) 2025-12-04T09:48:34.3550894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3551320Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3551723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3552225Z layer_outputs = layer_module( 2025-12-04T09:48:34.3552792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3553192Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3553611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3554020Z return func(*args, **kwargs) 2025-12-04T09:48:34.3554407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3554826Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3555241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3555656Z return func(*args, **kwargs) 2025-12-04T09:48:34.3556073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3556489Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3556938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3557500Z return func(*args, **kwargs) 2025-12-04T09:48:34.3557993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3558411Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3558634Z 2025-12-04T09:48:34.3558762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3559290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3559784Z res = mod(**inputs) 2025-12-04T09:48:34.3560175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3560857Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3561282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3561879Z layer_outputs = layer_module( 2025-12-04T09:48:34.3562324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3562801Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3563227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3563633Z return func(*args, **kwargs) 2025-12-04T09:48:34.3564017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3564432Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3564853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3565313Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3565750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.3566170Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.3566329Z 2025-12-04T09:48:34.3566448Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3566839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3567167Z res = mod(**inputs) 2025-12-04T09:48:34.3567523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3567915Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3568296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3568724Z layer_outputs = layer_module( 2025-12-04T09:48:34.3569123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3569499Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3569878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3570257Z return func(*args, **kwargs) 2025-12-04T09:48:34.3570622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3571025Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3571413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3571859Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3572281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.3572668Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.3572810Z 2025-12-04T09:48:34.3572918Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3573289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3573633Z res = mod(**inputs) 2025-12-04T09:48:34.3573986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3574402Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3574781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3575160Z layer_outputs = layer_module( 2025-12-04T09:48:34.3575513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3575890Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3576273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3576641Z return func(*args, **kwargs) 2025-12-04T09:48:34.3577007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3577400Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3577795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3578218Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3578635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.3579025Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.3579166Z 2025-12-04T09:48:34.3579282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3579642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3579975Z res = mod(**inputs) 2025-12-04T09:48:34.3580330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3580706Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3581087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3581469Z layer_outputs = layer_module( 2025-12-04T09:48:34.3581825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3582190Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3582576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3583013Z return func(*args, **kwargs) 2025-12-04T09:48:34.3583407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3583832Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3584257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3584658Z return func(*args, **kwargs) 2025-12-04T09:48:34.3585023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3585415Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3585820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3586255Z return func(*args, **kwargs) 2025-12-04T09:48:34.3586656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3587073Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3587221Z 2025-12-04T09:48:34.3587341Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3587726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3588080Z res = mod(**inputs) 2025-12-04T09:48:34.3588465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3588904Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3589298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3589709Z layer_outputs = layer_module( 2025-12-04T09:48:34.3590087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3590481Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3590894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3591306Z return func(*args, **kwargs) 2025-12-04T09:48:34.3591689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3592092Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3592502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3592914Z return func(*args, **kwargs) 2025-12-04T09:48:34.3593295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3593722Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3594135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3594532Z return func(*args, **kwargs) 2025-12-04T09:48:34.3594908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3595310Z key_states = self.k(current_states) 2025-12-04T09:48:34.3595455Z 2025-12-04T09:48:34.3595575Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3595962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3596305Z res = mod(**inputs) 2025-12-04T09:48:34.3596679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3597064Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3597434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3597859Z layer_outputs = layer_module( 2025-12-04T09:48:34.3598218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3598601Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3598998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3599399Z return func(*args, **kwargs) 2025-12-04T09:48:34.3599779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3600176Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3600744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3601198Z return func(*args, **kwargs) 2025-12-04T09:48:34.3601603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3602013Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3602409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3602792Z return func(*args, **kwargs) 2025-12-04T09:48:34.3603150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3603536Z value_states = self.v(current_states) 2025-12-04T09:48:34.3603705Z 2025-12-04T09:48:34.3603788Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3604008Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3604241Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3604611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3604946Z res = mod(**inputs) 2025-12-04T09:48:34.3605300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3605681Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3606056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3606431Z layer_outputs = layer_module( 2025-12-04T09:48:34.3606783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3607157Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3607547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3607929Z return func(*args, **kwargs) 2025-12-04T09:48:34.3608289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3608684Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3609081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3609455Z return func(*args, **kwargs) 2025-12-04T09:48:34.3609820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3610212Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3610600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3610984Z return func(*args, **kwargs) 2025-12-04T09:48:34.3611348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3611733Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3611871Z 2025-12-04T09:48:34.3611979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3612393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3612725Z res = mod(**inputs) 2025-12-04T09:48:34.3613085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3613461Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3613835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3614219Z layer_outputs = layer_module( 2025-12-04T09:48:34.3614567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3614945Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3615369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3615749Z return func(*args, **kwargs) 2025-12-04T09:48:34.3616106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3616508Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3616888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3617300Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3617696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.3618091Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.3618224Z 2025-12-04T09:48:34.3618334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3618687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3619007Z res = mod(**inputs) 2025-12-04T09:48:34.3619350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3619719Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3620081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3620458Z layer_outputs = layer_module( 2025-12-04T09:48:34.3620812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3621179Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3621566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3621945Z return func(*args, **kwargs) 2025-12-04T09:48:34.3622317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3622705Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3623102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3623527Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3623963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.3624373Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.3624531Z 2025-12-04T09:48:34.3624644Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3625036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3625377Z res = mod(**inputs) 2025-12-04T09:48:34.3625760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3626209Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3626605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3626978Z layer_outputs = layer_module( 2025-12-04T09:48:34.3627332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3627703Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3628082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3628463Z return func(*args, **kwargs) 2025-12-04T09:48:34.3628855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3629251Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3629654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3630085Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3630529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.3630948Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.3631097Z 2025-12-04T09:48:34.3631210Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3631601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3631971Z res = mod(**inputs) 2025-12-04T09:48:34.3632326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3632708Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3633082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3633464Z layer_outputs = layer_module( 2025-12-04T09:48:34.3633821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3634217Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3634625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3635016Z return func(*args, **kwargs) 2025-12-04T09:48:34.3635401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3635809Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3636222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3636623Z return func(*args, **kwargs) 2025-12-04T09:48:34.3637007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3637421Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3637833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3638234Z return func(*args, **kwargs) 2025-12-04T09:48:34.3638612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3639018Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3639166Z 2025-12-04T09:48:34.3639282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3639667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3640013Z res = mod(**inputs) 2025-12-04T09:48:34.3640382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3640974Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3641409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3641839Z layer_outputs = layer_module( 2025-12-04T09:48:34.3642253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3642669Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3643085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3643560Z return func(*args, **kwargs) 2025-12-04T09:48:34.3643914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3644302Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3644711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3645092Z return func(*args, **kwargs) 2025-12-04T09:48:34.3645478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3645892Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3646314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3646707Z return func(*args, **kwargs) 2025-12-04T09:48:34.3647279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3647709Z key_states = self.k(current_states) 2025-12-04T09:48:34.3647847Z 2025-12-04T09:48:34.3647964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3648331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3648668Z res = mod(**inputs) 2025-12-04T09:48:34.3649023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3649400Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3649778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3650170Z layer_outputs = layer_module( 2025-12-04T09:48:34.3650524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3650891Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3651275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3651654Z return func(*args, **kwargs) 2025-12-04T09:48:34.3652011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3652399Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3652770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3653126Z return func(*args, **kwargs) 2025-12-04T09:48:34.3653467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3653847Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3654223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3654589Z return func(*args, **kwargs) 2025-12-04T09:48:34.3654932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3655303Z value_states = self.v(current_states) 2025-12-04T09:48:34.3655438Z 2025-12-04T09:48:34.3655588Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3655831Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3656074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3656434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3656760Z res = mod(**inputs) 2025-12-04T09:48:34.3657103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3657482Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3657856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3658218Z layer_outputs = layer_module( 2025-12-04T09:48:34.3658568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3658957Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3659339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3659701Z return func(*args, **kwargs) 2025-12-04T09:48:34.3660059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3660441Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3660812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3661227Z return func(*args, **kwargs) 2025-12-04T09:48:34.3661576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3661957Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3662341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3662722Z return func(*args, **kwargs) 2025-12-04T09:48:34.3663097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3663462Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3663608Z 2025-12-04T09:48:34.3663715Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3664079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3664405Z res = mod(**inputs) 2025-12-04T09:48:34.3664744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3665121Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3665478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3665839Z layer_outputs = layer_module( 2025-12-04T09:48:34.3666176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3666532Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3666900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3667256Z return func(*args, **kwargs) 2025-12-04T09:48:34.3667606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3667987Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3668367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3668761Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3669163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.3669549Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.3669706Z 2025-12-04T09:48:34.3669817Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3670158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3670475Z res = mod(**inputs) 2025-12-04T09:48:34.3670817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3671182Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3671552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3671929Z layer_outputs = layer_module( 2025-12-04T09:48:34.3672282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3672660Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3673049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3673431Z return func(*args, **kwargs) 2025-12-04T09:48:34.3673786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3674186Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3674563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3674988Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3675383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.3675753Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.3675890Z 2025-12-04T09:48:34.3676006Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3676356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3676679Z res = mod(**inputs) 2025-12-04T09:48:34.3677018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3677387Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3677747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3678124Z layer_outputs = layer_module( 2025-12-04T09:48:34.3678481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3678852Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3679231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3679611Z return func(*args, **kwargs) 2025-12-04T09:48:34.3679974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3680380Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3680894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3681339Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3681784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.3682203Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.3682352Z 2025-12-04T09:48:34.3682460Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3682847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3683163Z res = mod(**inputs) 2025-12-04T09:48:34.3683559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3683942Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3684321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3684684Z layer_outputs = layer_module( 2025-12-04T09:48:34.3685032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3685396Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3685771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3686134Z return func(*args, **kwargs) 2025-12-04T09:48:34.3686509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3686906Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3687296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3687668Z return func(*args, **kwargs) 2025-12-04T09:48:34.3688024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3688413Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3688790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3689176Z return func(*args, **kwargs) 2025-12-04T09:48:34.3689523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3689887Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3690032Z 2025-12-04T09:48:34.3690141Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3690508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3690832Z res = mod(**inputs) 2025-12-04T09:48:34.3691172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3691550Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3691925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3692301Z layer_outputs = layer_module( 2025-12-04T09:48:34.3692647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3693019Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3693406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3693776Z return func(*args, **kwargs) 2025-12-04T09:48:34.3694139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3694524Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3694912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3695281Z return func(*args, **kwargs) 2025-12-04T09:48:34.3695644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3696032Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3696413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3696787Z return func(*args, **kwargs) 2025-12-04T09:48:34.3697149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3697567Z key_states = self.k(current_states) 2025-12-04T09:48:34.3697704Z 2025-12-04T09:48:34.3697812Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3698178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3698523Z res = mod(**inputs) 2025-12-04T09:48:34.3698868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3699249Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3699625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3699999Z layer_outputs = layer_module( 2025-12-04T09:48:34.3700363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3700738Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3701127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3701504Z return func(*args, **kwargs) 2025-12-04T09:48:34.3701863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3702243Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3702632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3703030Z return func(*args, **kwargs) 2025-12-04T09:48:34.3703394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3703785Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3704178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3704548Z return func(*args, **kwargs) 2025-12-04T09:48:34.3704924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3705297Z value_states = self.v(current_states) 2025-12-04T09:48:34.3705431Z 2025-12-04T09:48:34.3705514Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3705728Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3705964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3706322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3706636Z res = mod(**inputs) 2025-12-04T09:48:34.3706981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3707353Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3707714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3708085Z layer_outputs = layer_module( 2025-12-04T09:48:34.3708437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3708804Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3709188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3709574Z return func(*args, **kwargs) 2025-12-04T09:48:34.3709934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3710300Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3710681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3711051Z return func(*args, **kwargs) 2025-12-04T09:48:34.3711446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3711818Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3712199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3712570Z return func(*args, **kwargs) 2025-12-04T09:48:34.3712922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3713291Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3713429Z 2025-12-04T09:48:34.3713533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3713890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3714208Z res = mod(**inputs) 2025-12-04T09:48:34.3714570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3714954Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3715334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3715713Z layer_outputs = layer_module( 2025-12-04T09:48:34.3716076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3716457Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3716853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3717232Z return func(*args, **kwargs) 2025-12-04T09:48:34.3717595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3717983Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3718364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3718743Z return func(*args, **kwargs) 2025-12-04T09:48:34.3719104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 613, in forward 2025-12-04T09:48:34.3719539Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:48:34.3719728Z 2025-12-04T09:48:34.3719841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3720227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3720655Z res = mod(**inputs) 2025-12-04T09:48:34.3721030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3721454Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3721864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3722253Z layer_outputs = layer_module( 2025-12-04T09:48:34.3722629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3723028Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3723440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3723837Z return func(*args, **kwargs) 2025-12-04T09:48:34.3724229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3724658Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3725081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3725591Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3726115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.3726521Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.3726666Z 2025-12-04T09:48:34.3726790Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3727177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3727526Z res = mod(**inputs) 2025-12-04T09:48:34.3727904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3728311Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3728713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3729139Z layer_outputs = layer_module( 2025-12-04T09:48:34.3729524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3729914Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3730324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3730727Z return func(*args, **kwargs) 2025-12-04T09:48:34.3731102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3731505Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3731949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3732398Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3732836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.3733248Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.3733397Z 2025-12-04T09:48:34.3733519Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3733906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3734248Z res = mod(**inputs) 2025-12-04T09:48:34.3734617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3735019Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3735408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3735810Z layer_outputs = layer_module( 2025-12-04T09:48:34.3736181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3736551Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3736936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3737315Z return func(*args, **kwargs) 2025-12-04T09:48:34.3737681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3738065Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3738461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3738883Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3739300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.3739676Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.3739821Z 2025-12-04T09:48:34.3739930Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3740355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3740717Z res = mod(**inputs) 2025-12-04T09:48:34.3741092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3741505Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3741910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3742310Z layer_outputs = layer_module( 2025-12-04T09:48:34.3742700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3743093Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3743504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3743911Z return func(*args, **kwargs) 2025-12-04T09:48:34.3744274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3744720Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3745106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3745491Z return func(*args, **kwargs) 2025-12-04T09:48:34.3745847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3746255Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3746627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3746995Z return func(*args, **kwargs) 2025-12-04T09:48:34.3747482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3747862Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3748000Z 2025-12-04T09:48:34.3748109Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3748470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3748791Z res = mod(**inputs) 2025-12-04T09:48:34.3749132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3749507Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3749883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3750273Z layer_outputs = layer_module( 2025-12-04T09:48:34.3750623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3750994Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3751378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3751749Z return func(*args, **kwargs) 2025-12-04T09:48:34.3752120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3752494Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3752871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3753232Z return func(*args, **kwargs) 2025-12-04T09:48:34.3753581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3753958Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3754333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3754749Z return func(*args, **kwargs) 2025-12-04T09:48:34.3755126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3755501Z key_states = self.k(current_states) 2025-12-04T09:48:34.3755636Z 2025-12-04T09:48:34.3755741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3756104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3756426Z res = mod(**inputs) 2025-12-04T09:48:34.3756775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3757144Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3757519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3757985Z layer_outputs = layer_module( 2025-12-04T09:48:34.3758333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3758701Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3759081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3759450Z return func(*args, **kwargs) 2025-12-04T09:48:34.3759802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3760213Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3760683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3761085Z return func(*args, **kwargs) 2025-12-04T09:48:34.3761476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3761882Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3762280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3762655Z return func(*args, **kwargs) 2025-12-04T09:48:34.3763018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3763395Z value_states = self.v(current_states) 2025-12-04T09:48:34.3763528Z 2025-12-04T09:48:34.3763617Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3763826Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3764057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3764405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3764710Z res = mod(**inputs) 2025-12-04T09:48:34.3765066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3765452Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3765820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3766196Z layer_outputs = layer_module( 2025-12-04T09:48:34.3766553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3766924Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3767285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3767650Z return func(*args, **kwargs) 2025-12-04T09:48:34.3767997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3768359Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3768771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3769162Z return func(*args, **kwargs) 2025-12-04T09:48:34.3769515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3769890Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3770264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3770631Z return func(*args, **kwargs) 2025-12-04T09:48:34.3770986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3771352Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3771490Z 2025-12-04T09:48:34.3771594Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3771973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3772295Z res = mod(**inputs) 2025-12-04T09:48:34.3772630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3772990Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3773343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3773692Z layer_outputs = layer_module( 2025-12-04T09:48:34.3774033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3774426Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3774792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3775158Z return func(*args, **kwargs) 2025-12-04T09:48:34.3775510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3775896Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3776266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3776680Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3777077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.3791278Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.3791466Z 2025-12-04T09:48:34.3791593Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3791973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3792310Z res = mod(**inputs) 2025-12-04T09:48:34.3792708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3793107Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3793485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3793866Z layer_outputs = layer_module( 2025-12-04T09:48:34.3794225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3794613Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3795092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3795486Z return func(*args, **kwargs) 2025-12-04T09:48:34.3795864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3796266Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3796780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3797285Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3797709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.3798146Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.3798308Z 2025-12-04T09:48:34.3798422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3798826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3799193Z res = mod(**inputs) 2025-12-04T09:48:34.3799579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3799988Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3800517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3800962Z layer_outputs = layer_module( 2025-12-04T09:48:34.3801366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3801781Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3802213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3802609Z return func(*args, **kwargs) 2025-12-04T09:48:34.3803005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3803392Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3803774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3804178Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3804591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.3804964Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.3805104Z 2025-12-04T09:48:34.3805213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3805592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3805912Z res = mod(**inputs) 2025-12-04T09:48:34.3806259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3806624Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3807068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3807486Z layer_outputs = layer_module( 2025-12-04T09:48:34.3807835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3808207Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3808589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3808969Z return func(*args, **kwargs) 2025-12-04T09:48:34.3809323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3809708Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3810099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3810470Z return func(*args, **kwargs) 2025-12-04T09:48:34.3810823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3811212Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3811636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3812001Z return func(*args, **kwargs) 2025-12-04T09:48:34.3812407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3812787Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3812924Z 2025-12-04T09:48:34.3813037Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3813391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3813714Z res = mod(**inputs) 2025-12-04T09:48:34.3814057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3814425Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3815364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3815751Z layer_outputs = layer_module( 2025-12-04T09:48:34.3816097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3816456Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3816835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3817204Z return func(*args, **kwargs) 2025-12-04T09:48:34.3817573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3817951Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3818328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3818697Z return func(*args, **kwargs) 2025-12-04T09:48:34.3819044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3819421Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3819802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3820169Z return func(*args, **kwargs) 2025-12-04T09:48:34.3820514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3820892Z key_states = self.k(current_states) 2025-12-04T09:48:34.3821025Z 2025-12-04T09:48:34.3821135Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3821484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3821804Z res = mod(**inputs) 2025-12-04T09:48:34.3822155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3822516Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3822863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3823229Z layer_outputs = layer_module( 2025-12-04T09:48:34.3823576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3823928Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3824312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3824708Z return func(*args, **kwargs) 2025-12-04T09:48:34.3825058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3825427Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3825824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3826217Z return func(*args, **kwargs) 2025-12-04T09:48:34.3826557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3826936Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3827319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3827689Z return func(*args, **kwargs) 2025-12-04T09:48:34.3828037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3828420Z value_states = self.v(current_states) 2025-12-04T09:48:34.3828549Z 2025-12-04T09:48:34.3828638Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3828867Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3829099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3829458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3829776Z res = mod(**inputs) 2025-12-04T09:48:34.3830114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3830492Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3830865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3831255Z layer_outputs = layer_module( 2025-12-04T09:48:34.3831606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3831976Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3832369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3832731Z return func(*args, **kwargs) 2025-12-04T09:48:34.3833084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3833454Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3833832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3834194Z return func(*args, **kwargs) 2025-12-04T09:48:34.3834537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3834904Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3835277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3835645Z return func(*args, **kwargs) 2025-12-04T09:48:34.3836009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3836395Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3836531Z 2025-12-04T09:48:34.3836636Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3836999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3837325Z res = mod(**inputs) 2025-12-04T09:48:34.3837667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3838051Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3838420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3838795Z layer_outputs = layer_module( 2025-12-04T09:48:34.3839146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3839536Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3839946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3840343Z return func(*args, **kwargs) 2025-12-04T09:48:34.3840820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3841239Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3841667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3842084Z return func(*args, **kwargs) 2025-12-04T09:48:34.3842485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3842897Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3843348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3843748Z return func(*args, **kwargs) 2025-12-04T09:48:34.3844143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3844530Z key_states = self.k(current_states) 2025-12-04T09:48:34.3844664Z 2025-12-04T09:48:34.3844777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3845139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3845491Z res = mod(**inputs) 2025-12-04T09:48:34.3845848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3846224Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3846603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3846986Z layer_outputs = layer_module( 2025-12-04T09:48:34.3847487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3847857Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3848250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3848631Z return func(*args, **kwargs) 2025-12-04T09:48:34.3848994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3849386Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3849782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3850166Z return func(*args, **kwargs) 2025-12-04T09:48:34.3850527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3850921Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3851313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3851685Z return func(*args, **kwargs) 2025-12-04T09:48:34.3852047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3852431Z value_states = self.v(current_states) 2025-12-04T09:48:34.3852571Z 2025-12-04T09:48:34.3852659Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3852870Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3853116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3853480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3853804Z res = mod(**inputs) 2025-12-04T09:48:34.3854234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3854667Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3855070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3855459Z layer_outputs = layer_module( 2025-12-04T09:48:34.3855818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3856194Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3856575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3856964Z return func(*args, **kwargs) 2025-12-04T09:48:34.3857340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3857717Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3858090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3858462Z return func(*args, **kwargs) 2025-12-04T09:48:34.3858814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3859194Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3859568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3859959Z return func(*args, **kwargs) 2025-12-04T09:48:34.3860313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3860687Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3860832Z 2025-12-04T09:48:34.3860942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3861310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3861641Z res = mod(**inputs) 2025-12-04T09:48:34.3861984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3862372Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3862753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3863124Z layer_outputs = layer_module( 2025-12-04T09:48:34.3863485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3863858Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3864247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3864622Z return func(*args, **kwargs) 2025-12-04T09:48:34.3864989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3865375Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3865766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3866137Z return func(*args, **kwargs) 2025-12-04T09:48:34.3866499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3866901Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3867283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3867663Z return func(*args, **kwargs) 2025-12-04T09:48:34.3868028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3868432Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3868590Z 2025-12-04T09:48:34.3868700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3869071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3869424Z res = mod(**inputs) 2025-12-04T09:48:34.3869758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3870133Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3870508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3870881Z layer_outputs = layer_module( 2025-12-04T09:48:34.3871227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3871615Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3872002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3872378Z return func(*args, **kwargs) 2025-12-04T09:48:34.3872733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3873115Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3873501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3873887Z return func(*args, **kwargs) 2025-12-04T09:48:34.3874249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3874635Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3875026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3875396Z return func(*args, **kwargs) 2025-12-04T09:48:34.3875757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3876184Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3876316Z 2025-12-04T09:48:34.3876428Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3876787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3877115Z res = mod(**inputs) 2025-12-04T09:48:34.3877468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3877844Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3878220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3878601Z layer_outputs = layer_module( 2025-12-04T09:48:34.3878960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3879323Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3879708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3880104Z return func(*args, **kwargs) 2025-12-04T09:48:34.3880555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3880977Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3881411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3881841Z return func(*args, **kwargs) 2025-12-04T09:48:34.3882243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 613, in forward 2025-12-04T09:48:34.3882750Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:48:34.3882958Z 2025-12-04T09:48:34.3883079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3883443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3883774Z res = mod(**inputs) 2025-12-04T09:48:34.3884124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3884506Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3884875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3885252Z layer_outputs = layer_module( 2025-12-04T09:48:34.3885615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3886019Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3886400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3886781Z return func(*args, **kwargs) 2025-12-04T09:48:34.3887144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3887534Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3887931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3888370Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3888790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.3889169Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.3889315Z 2025-12-04T09:48:34.3889424Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3889795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3890114Z res = mod(**inputs) 2025-12-04T09:48:34.3890466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3890842Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3891213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3891583Z layer_outputs = layer_module( 2025-12-04T09:48:34.3891938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3892311Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3892767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3893131Z return func(*args, **kwargs) 2025-12-04T09:48:34.3893487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3893883Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3894275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3894688Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3895088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.3895464Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.3895598Z 2025-12-04T09:48:34.3895702Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3896059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3896381Z res = mod(**inputs) 2025-12-04T09:48:34.3896771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:48:34.3897188Z encoder_outputs = self.encoder( 2025-12-04T09:48:34.3897563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3897940Z layer_outputs = layer_module( 2025-12-04T09:48:34.3898288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3898662Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3899047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3899424Z return func(*args, **kwargs) 2025-12-04T09:48:34.3899800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3900192Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3900577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3900986Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3901411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.3901800Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.3901937Z 2025-12-04T09:48:34.3902073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3902432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3902763Z res = mod(**inputs) 2025-12-04T09:48:34.3903117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3903494Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3903875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3904255Z layer_outputs = layer_module( 2025-12-04T09:48:34.3904600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3904955Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3905343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3905728Z return func(*args, **kwargs) 2025-12-04T09:48:34.3906092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3906473Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3906859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3907237Z return func(*args, **kwargs) 2025-12-04T09:48:34.3907595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3907985Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3908376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3908757Z return func(*args, **kwargs) 2025-12-04T09:48:34.3909126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3909529Z key_states = self.k(current_states) 2025-12-04T09:48:34.3909672Z 2025-12-04T09:48:34.3909792Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3910173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3910518Z res = mod(**inputs) 2025-12-04T09:48:34.3910927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3911329Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3911720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3912127Z layer_outputs = layer_module( 2025-12-04T09:48:34.3912480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3912845Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3913232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3913611Z return func(*args, **kwargs) 2025-12-04T09:48:34.3913992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3914395Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3914809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3915194Z return func(*args, **kwargs) 2025-12-04T09:48:34.3915555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3915989Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3916402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3916823Z return func(*args, **kwargs) 2025-12-04T09:48:34.3917200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3917293Z value_states = self.v(current_states) 2025-12-04T09:48:34.3917297Z 2025-12-04T09:48:34.3917385Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3917481Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3917595Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3917811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3917888Z res = mod(**inputs) 2025-12-04T09:48:34.3918146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3918224Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3918495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3918571Z layer_outputs = layer_module( 2025-12-04T09:48:34.3918821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3918909Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3919171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3919255Z return func(*args, **kwargs) 2025-12-04T09:48:34.3919509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3919595Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3919864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3919940Z return func(*args, **kwargs) 2025-12-04T09:48:34.3920199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3920290Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3920650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3920770Z return func(*args, **kwargs) 2025-12-04T09:48:34.3921059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3921155Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3921159Z 2025-12-04T09:48:34.3921277Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3921508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3921585Z res = mod(**inputs) 2025-12-04T09:48:34.3921841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3921922Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3922189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3922281Z layer_outputs = layer_module( 2025-12-04T09:48:34.3922519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3922600Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3922845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3922924Z return func(*args, **kwargs) 2025-12-04T09:48:34.3923163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3923255Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3923520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3923640Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3923883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.3923965Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.3923969Z 2025-12-04T09:48:34.3924073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3924280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3924346Z res = mod(**inputs) 2025-12-04T09:48:34.3924593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3924668Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3924908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3924988Z layer_outputs = layer_module( 2025-12-04T09:48:34.3925215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3925295Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3925550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3925620Z return func(*args, **kwargs) 2025-12-04T09:48:34.3925862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3925954Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3926191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3926316Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3926552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.3926642Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.3926645Z 2025-12-04T09:48:34.3926751Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3926985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3927059Z res = mod(**inputs) 2025-12-04T09:48:34.3927297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3927372Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3927614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3927688Z layer_outputs = layer_module( 2025-12-04T09:48:34.3927922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3928001Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3928246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3928340Z return func(*args, **kwargs) 2025-12-04T09:48:34.3928582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3928675Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3928918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3929035Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3929284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.3929385Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.3929389Z 2025-12-04T09:48:34.3929501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3929698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3929773Z res = mod(**inputs) 2025-12-04T09:48:34.3930023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3930097Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3930346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3930417Z layer_outputs = layer_module( 2025-12-04T09:48:34.3930648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3930741Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3931004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3931087Z return func(*args, **kwargs) 2025-12-04T09:48:34.3931356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3931444Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3931731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3931804Z return func(*args, **kwargs) 2025-12-04T09:48:34.3932077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3932168Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3932433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3932516Z return func(*args, **kwargs) 2025-12-04T09:48:34.3932786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3932869Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3932873Z 2025-12-04T09:48:34.3932997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3933262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3933342Z res = mod(**inputs) 2025-12-04T09:48:34.3933607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3933685Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3933948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3934024Z layer_outputs = layer_module( 2025-12-04T09:48:34.3934269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3934362Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3934631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3934731Z return func(*args, **kwargs) 2025-12-04T09:48:34.3934989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3935079Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3935350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3935423Z return func(*args, **kwargs) 2025-12-04T09:48:34.3935684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3935789Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3936047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3936136Z return func(*args, **kwargs) 2025-12-04T09:48:34.3936374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3936452Z key_states = self.k(current_states) 2025-12-04T09:48:34.3936464Z 2025-12-04T09:48:34.3936569Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3936766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3936838Z res = mod(**inputs) 2025-12-04T09:48:34.3937078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3937153Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3937400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3937476Z layer_outputs = layer_module( 2025-12-04T09:48:34.3937720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3937806Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3938070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3938149Z return func(*args, **kwargs) 2025-12-04T09:48:34.3938400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3938486Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3938752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3938826Z return func(*args, **kwargs) 2025-12-04T09:48:34.3939085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3939174Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3939430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3939551Z return func(*args, **kwargs) 2025-12-04T09:48:34.3939809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3939888Z value_states = self.v(current_states) 2025-12-04T09:48:34.3939899Z 2025-12-04T09:48:34.3939980Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3940061Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3940176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3940377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3940445Z res = mod(**inputs) 2025-12-04T09:48:34.3940694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3940770Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3941024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3941113Z layer_outputs = layer_module( 2025-12-04T09:48:34.3941349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3941441Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3941752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3941822Z return func(*args, **kwargs) 2025-12-04T09:48:34.3942084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3942168Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3942417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3942487Z return func(*args, **kwargs) 2025-12-04T09:48:34.3942728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3942822Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3943080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3943153Z return func(*args, **kwargs) 2025-12-04T09:48:34.3943413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3943507Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3943510Z 2025-12-04T09:48:34.3943621Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3943818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3943883Z res = mod(**inputs) 2025-12-04T09:48:34.3944131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3944210Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3944456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3944540Z layer_outputs = layer_module( 2025-12-04T09:48:34.3944775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3944868Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3945124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3945197Z return func(*args, **kwargs) 2025-12-04T09:48:34.3945455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3945541Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3945822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3945917Z return func(*args, **kwargs) 2025-12-04T09:48:34.3946167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3946265Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3946512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3946581Z return func(*args, **kwargs) 2025-12-04T09:48:34.3946820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3946898Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3946902Z 2025-12-04T09:48:34.3947013Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3947459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3947533Z res = mod(**inputs) 2025-12-04T09:48:34.3947788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3947863Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3948101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3948184Z layer_outputs = layer_module( 2025-12-04T09:48:34.3948414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3948540Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3948790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3948862Z return func(*args, **kwargs) 2025-12-04T09:48:34.3949122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3949211Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3949474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3949548Z return func(*args, **kwargs) 2025-12-04T09:48:34.3949799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3949897Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3950152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3950225Z return func(*args, **kwargs) 2025-12-04T09:48:34.3950480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3950564Z key_states = self.k(current_states) 2025-12-04T09:48:34.3950569Z 2025-12-04T09:48:34.3950687Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3950896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3950963Z res = mod(**inputs) 2025-12-04T09:48:34.3951222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3951298Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3951549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3951636Z layer_outputs = layer_module( 2025-12-04T09:48:34.3951873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3951964Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3952223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3952346Z return func(*args, **kwargs) 2025-12-04T09:48:34.3952606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3952692Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3952953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3953025Z return func(*args, **kwargs) 2025-12-04T09:48:34.3953278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3953375Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3953632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3953720Z return func(*args, **kwargs) 2025-12-04T09:48:34.3953985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3954067Z value_states = self.v(current_states) 2025-12-04T09:48:34.3954071Z 2025-12-04T09:48:34.3954166Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3954252Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3954363Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3954584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3954685Z res = mod(**inputs) 2025-12-04T09:48:34.3954942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3955026Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3955283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3955367Z layer_outputs = layer_module( 2025-12-04T09:48:34.3955608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3955690Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3955958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3956031Z return func(*args, **kwargs) 2025-12-04T09:48:34.3956287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3956384Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3956644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3956723Z return func(*args, **kwargs) 2025-12-04T09:48:34.3956977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3957071Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3957336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3957411Z return func(*args, **kwargs) 2025-12-04T09:48:34.3957668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3957749Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3957754Z 2025-12-04T09:48:34.3957863Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3958081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3958150Z res = mod(**inputs) 2025-12-04T09:48:34.3958406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3958511Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3958784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3958870Z layer_outputs = layer_module( 2025-12-04T09:48:34.3959116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3959198Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3959468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3959543Z return func(*args, **kwargs) 2025-12-04T09:48:34.3959809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3959907Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3960183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3960327Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3960661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.3960754Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.3960758Z 2025-12-04T09:48:34.3960882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3961100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3961207Z res = mod(**inputs) 2025-12-04T09:48:34.3961470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3961553Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3961822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3961895Z layer_outputs = layer_module( 2025-12-04T09:48:34.3962115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3962203Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3962440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3962516Z return func(*args, **kwargs) 2025-12-04T09:48:34.3962746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3962837Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3963076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3963191Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3963429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.3963517Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.3963520Z 2025-12-04T09:48:34.3963622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3963832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3963897Z res = mod(**inputs) 2025-12-04T09:48:34.3964138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3964221Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3964458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3964538Z layer_outputs = layer_module( 2025-12-04T09:48:34.3964765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3964911Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3965197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3965272Z return func(*args, **kwargs) 2025-12-04T09:48:34.3965535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3965627Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3965865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3965990Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3966225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.3966329Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.3966334Z 2025-12-04T09:48:34.3966640Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3966851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3966922Z res = mod(**inputs) 2025-12-04T09:48:34.3967159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3967233Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3967477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3967573Z layer_outputs = layer_module( 2025-12-04T09:48:34.3967804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3967883Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3968126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3968205Z return func(*args, **kwargs) 2025-12-04T09:48:34.3968442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3968524Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3968770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3968837Z return func(*args, **kwargs) 2025-12-04T09:48:34.3969075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3969159Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3969400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3969477Z return func(*args, **kwargs) 2025-12-04T09:48:34.3969713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3969793Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3969802Z 2025-12-04T09:48:34.3969902Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3970100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3970172Z res = mod(**inputs) 2025-12-04T09:48:34.3970406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3970480Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3970722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3970793Z layer_outputs = layer_module( 2025-12-04T09:48:34.3971022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3971119Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3971375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3971454Z return func(*args, **kwargs) 2025-12-04T09:48:34.3971687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3971766Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3972015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3972083Z return func(*args, **kwargs) 2025-12-04T09:48:34.3972327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3972408Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3972666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3972749Z return func(*args, **kwargs) 2025-12-04T09:48:34.3972988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3973065Z key_states = self.k(current_states) 2025-12-04T09:48:34.3973076Z 2025-12-04T09:48:34.3973180Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3973375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3973467Z res = mod(**inputs) 2025-12-04T09:48:34.3973702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3973773Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3974015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3974088Z layer_outputs = layer_module( 2025-12-04T09:48:34.3974317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3974396Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3974638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3974713Z return func(*args, **kwargs) 2025-12-04T09:48:34.3974948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3975028Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3975275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3975342Z return func(*args, **kwargs) 2025-12-04T09:48:34.3975593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3975677Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3975913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3975986Z return func(*args, **kwargs) 2025-12-04T09:48:34.3976222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3976299Z value_states = self.v(current_states) 2025-12-04T09:48:34.3976310Z 2025-12-04T09:48:34.3976388Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3976467Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3976574Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3976769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3976834Z res = mod(**inputs) 2025-12-04T09:48:34.3977104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3977178Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3977418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3977490Z layer_outputs = layer_module( 2025-12-04T09:48:34.3977709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3977799Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3978035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3978101Z return func(*args, **kwargs) 2025-12-04T09:48:34.3978366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3978450Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3978696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3978764Z return func(*args, **kwargs) 2025-12-04T09:48:34.3978993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.3979084Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.3979326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3979409Z return func(*args, **kwargs) 2025-12-04T09:48:34.3979652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3979728Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3979732Z 2025-12-04T09:48:34.3979843Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3980041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3980104Z res = mod(**inputs) 2025-12-04T09:48:34.3980348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3980417Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3980658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3980728Z layer_outputs = layer_module( 2025-12-04T09:48:34.3980949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3981033Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3981272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3981339Z return func(*args, **kwargs) 2025-12-04T09:48:34.3981586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.3981664Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.3981909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3981976Z return func(*args, **kwargs) 2025-12-04T09:48:34.3982214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 613, in forward 2025-12-04T09:48:34.3982360Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:48:34.3982365Z 2025-12-04T09:48:34.3982466Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3982684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3982749Z res = mod(**inputs) 2025-12-04T09:48:34.3983019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3983099Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3983333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3983403Z layer_outputs = layer_module( 2025-12-04T09:48:34.3983629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3983710Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3983957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3984026Z return func(*args, **kwargs) 2025-12-04T09:48:34.3984281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3984375Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3984634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3984708Z return func(*args, **kwargs) 2025-12-04T09:48:34.3984965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3985056Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3985324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3985411Z return func(*args, **kwargs) 2025-12-04T09:48:34.3985649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.3985735Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.3985738Z 2025-12-04T09:48:34.3985844Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3986052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3986118Z res = mod(**inputs) 2025-12-04T09:48:34.3986355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3986437Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3986675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3986750Z layer_outputs = layer_module( 2025-12-04T09:48:34.3986983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3987061Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3987309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3987381Z return func(*args, **kwargs) 2025-12-04T09:48:34.3987621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3987709Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3987951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3988021Z return func(*args, **kwargs) 2025-12-04T09:48:34.3988275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3988368Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3988633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3988705Z return func(*args, **kwargs) 2025-12-04T09:48:34.3988967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.3989097Z key_states = self.k(current_states) 2025-12-04T09:48:34.3989101Z 2025-12-04T09:48:34.3989210Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3989430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3989500Z res = mod(**inputs) 2025-12-04T09:48:34.3989758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3989845Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3990098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3990175Z layer_outputs = layer_module( 2025-12-04T09:48:34.3990444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3990529Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3990795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3990868Z return func(*args, **kwargs) 2025-12-04T09:48:34.3991123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3991217Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3991476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3991568Z return func(*args, **kwargs) 2025-12-04T09:48:34.3991827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3991915Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3992176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3992251Z return func(*args, **kwargs) 2025-12-04T09:48:34.3992502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.3992592Z value_states = self.v(current_states) 2025-12-04T09:48:34.3992596Z 2025-12-04T09:48:34.3992680Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3992763Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.3992879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3993090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3993166Z res = mod(**inputs) 2025-12-04T09:48:34.3993420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3993500Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3993759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3993840Z layer_outputs = layer_module( 2025-12-04T09:48:34.3994085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3994166Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3994421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3994501Z return func(*args, **kwargs) 2025-12-04T09:48:34.3994754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.3994840Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.3995103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3995177Z return func(*args, **kwargs) 2025-12-04T09:48:34.3995484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.3995575Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.3995833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3995914Z return func(*args, **kwargs) 2025-12-04T09:48:34.3996168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.3996253Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.3996266Z 2025-12-04T09:48:34.3996374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3996585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3996660Z res = mod(**inputs) 2025-12-04T09:48:34.3996933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3997016Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.3997271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.3997347Z layer_outputs = layer_module( 2025-12-04T09:48:34.3997594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.3997675Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.3997957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.3998039Z return func(*args, **kwargs) 2025-12-04T09:48:34.3998296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.3998396Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.3998665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.3998792Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.3999058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.3999142Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.3999146Z 2025-12-04T09:48:34.3999256Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.3999481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.3999549Z res = mod(**inputs) 2025-12-04T09:48:34.3999808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.3999893Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4000162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4000251Z layer_outputs = layer_module( 2025-12-04T09:48:34.4000575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4000665Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4000934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4001009Z return func(*args, **kwargs) 2025-12-04T09:48:34.4001280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4001381Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4001658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.4001795Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.4002120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.4002209Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.4002221Z 2025-12-04T09:48:34.4002335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4002548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4002625Z res = mod(**inputs) 2025-12-04T09:48:34.4002885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4002964Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4003235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4003329Z layer_outputs = layer_module( 2025-12-04T09:48:34.4003578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4003663Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4003924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4004004Z return func(*args, **kwargs) 2025-12-04T09:48:34.4004257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4004373Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4004635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.4004759Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.4005022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.4005107Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.4005112Z 2025-12-04T09:48:34.4005223Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4005446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4005514Z res = mod(**inputs) 2025-12-04T09:48:34.4005778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4005859Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4006105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4006184Z layer_outputs = layer_module( 2025-12-04T09:48:34.4006411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4006491Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4006751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4006820Z return func(*args, **kwargs) 2025-12-04T09:48:34.4007067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4007149Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4007398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4007478Z return func(*args, **kwargs) 2025-12-04T09:48:34.4007718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4007801Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4008056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4008147Z return func(*args, **kwargs) 2025-12-04T09:48:34.4008416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.4008497Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.4008500Z 2025-12-04T09:48:34.4008605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4008808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4008874Z res = mod(**inputs) 2025-12-04T09:48:34.4009118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4009192Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4009429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4009529Z layer_outputs = layer_module( 2025-12-04T09:48:34.4009759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4009838Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4010088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4010159Z return func(*args, **kwargs) 2025-12-04T09:48:34.4010400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4010502Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4010748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4010829Z return func(*args, **kwargs) 2025-12-04T09:48:34.4011073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4011166Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4011439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4011515Z return func(*args, **kwargs) 2025-12-04T09:48:34.4011777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.4011861Z key_states = self.k(current_states) 2025-12-04T09:48:34.4011865Z 2025-12-04T09:48:34.4011978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4012201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4012272Z res = mod(**inputs) 2025-12-04T09:48:34.4012537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4012615Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4012858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4012943Z layer_outputs = layer_module( 2025-12-04T09:48:34.4013185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4013272Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4013540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4013619Z return func(*args, **kwargs) 2025-12-04T09:48:34.4013884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4013975Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4014242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4014323Z return func(*args, **kwargs) 2025-12-04T09:48:34.4014598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4014682Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4014934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4015002Z return func(*args, **kwargs) 2025-12-04T09:48:34.4015246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.4015327Z value_states = self.v(current_states) 2025-12-04T09:48:34.4015330Z 2025-12-04T09:48:34.4015411Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4015498Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4015603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4015822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4015898Z res = mod(**inputs) 2025-12-04T09:48:34.4016139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4016221Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4016458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4016531Z layer_outputs = layer_module( 2025-12-04T09:48:34.4016764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4016865Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4017119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4017193Z return func(*args, **kwargs) 2025-12-04T09:48:34.4017434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4017528Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4017773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4017847Z return func(*args, **kwargs) 2025-12-04T09:48:34.4018095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4018181Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4018438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4018510Z return func(*args, **kwargs) 2025-12-04T09:48:34.4018748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.4018839Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.4018845Z 2025-12-04T09:48:34.4018954Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4019159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4019236Z res = mod(**inputs) 2025-12-04T09:48:34.4019477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4019560Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4019801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4019877Z layer_outputs = layer_module( 2025-12-04T09:48:34.4020113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4020195Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4020447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4020556Z return func(*args, **kwargs) 2025-12-04T09:48:34.4020799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4020890Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4021131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4021200Z return func(*args, **kwargs) 2025-12-04T09:48:34.4021451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4021538Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4021793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4021883Z return func(*args, **kwargs) 2025-12-04T09:48:34.4022122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.4022210Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.4022213Z 2025-12-04T09:48:34.4022318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4022520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4022597Z res = mod(**inputs) 2025-12-04T09:48:34.4022838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4022940Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4023191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4023261Z layer_outputs = layer_module( 2025-12-04T09:48:34.4023493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4023573Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4023822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4023889Z return func(*args, **kwargs) 2025-12-04T09:48:34.4024124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4024212Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4024458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4024526Z return func(*args, **kwargs) 2025-12-04T09:48:34.4024775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4024861Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4025117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4025185Z return func(*args, **kwargs) 2025-12-04T09:48:34.4025428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.4025512Z key_states = self.k(current_states) 2025-12-04T09:48:34.4025516Z 2025-12-04T09:48:34.4025620Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4025826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4025894Z res = mod(**inputs) 2025-12-04T09:48:34.4026138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4026219Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4026464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4026585Z layer_outputs = layer_module( 2025-12-04T09:48:34.4026820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4026899Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4027147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4027215Z return func(*args, **kwargs) 2025-12-04T09:48:34.4027449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4027539Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4027779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4027865Z return func(*args, **kwargs) 2025-12-04T09:48:34.4028113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4028201Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4028450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4028518Z return func(*args, **kwargs) 2025-12-04T09:48:34.4028768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.4028876Z value_states = self.v(current_states) 2025-12-04T09:48:34.4028880Z 2025-12-04T09:48:34.4028967Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4029052Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4029168Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4029377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4029454Z res = mod(**inputs) 2025-12-04T09:48:34.4029711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4029791Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4030048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4030124Z layer_outputs = layer_module( 2025-12-04T09:48:34.4030361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4030457Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4030712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4030793Z return func(*args, **kwargs) 2025-12-04T09:48:34.4031044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4031132Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4031400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4031475Z return func(*args, **kwargs) 2025-12-04T09:48:34.4031733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4031822Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4032079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4032156Z return func(*args, **kwargs) 2025-12-04T09:48:34.4032393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.4032470Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.4032483Z 2025-12-04T09:48:34.4032587Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4032826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4032902Z res = mod(**inputs) 2025-12-04T09:48:34.4033143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4033219Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4033464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4033539Z layer_outputs = layer_module( 2025-12-04T09:48:34.4033771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4033848Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4034105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4034185Z return func(*args, **kwargs) 2025-12-04T09:48:34.4034423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4034506Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4034758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4034828Z return func(*args, **kwargs) 2025-12-04T09:48:34.4035071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 652, in forward 2025-12-04T09:48:34.4035226Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:48:34.4035229Z 2025-12-04T09:48:34.4035335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4035547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4035613Z res = mod(**inputs) 2025-12-04T09:48:34.4035856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4035938Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4036175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4036254Z layer_outputs = layer_module( 2025-12-04T09:48:34.4036474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4036556Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4036807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4036877Z return func(*args, **kwargs) 2025-12-04T09:48:34.4037122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4037217Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4037465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.4037599Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.4037893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.4037977Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.4037982Z 2025-12-04T09:48:34.4038099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4038309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4038390Z res = mod(**inputs) 2025-12-04T09:48:34.4038644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4038777Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4039066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4039145Z layer_outputs = layer_module( 2025-12-04T09:48:34.4039385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4039475Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4039736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4039818Z return func(*args, **kwargs) 2025-12-04T09:48:34.4040072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4040168Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4040529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.4040673Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.4040961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.4041052Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.4041057Z 2025-12-04T09:48:34.4041172Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4041402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4041495Z res = mod(**inputs) 2025-12-04T09:48:34.4041779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4041867Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4042134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4042219Z layer_outputs = layer_module( 2025-12-04T09:48:34.4042462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4042546Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4042817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4042890Z return func(*args, **kwargs) 2025-12-04T09:48:34.4043159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4043252Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4043491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.4043616Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.4043860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.4043944Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.4043949Z 2025-12-04T09:48:34.4044059Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4044258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4044329Z res = mod(**inputs) 2025-12-04T09:48:34.4044572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4044646Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4044893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4044965Z layer_outputs = layer_module( 2025-12-04T09:48:34.4045193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4045302Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4045573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4045655Z return func(*args, **kwargs) 2025-12-04T09:48:34.4045891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4045973Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4046224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4046296Z return func(*args, **kwargs) 2025-12-04T09:48:34.4046538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4046623Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4046890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4046972Z return func(*args, **kwargs) 2025-12-04T09:48:34.4047362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.4047448Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.4047462Z 2025-12-04T09:48:34.4047569Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4047769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4047901Z res = mod(**inputs) 2025-12-04T09:48:34.4048152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4048226Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4048478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4048552Z layer_outputs = layer_module( 2025-12-04T09:48:34.4048791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4048872Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4049118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4049194Z return func(*args, **kwargs) 2025-12-04T09:48:34.4049445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4049528Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4049787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4049859Z return func(*args, **kwargs) 2025-12-04T09:48:34.4050111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4050199Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4050464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4050545Z return func(*args, **kwargs) 2025-12-04T09:48:34.4050802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.4050883Z key_states = self.k(current_states) 2025-12-04T09:48:34.4050908Z 2025-12-04T09:48:34.4051014Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4051216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4051288Z res = mod(**inputs) 2025-12-04T09:48:34.4051539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4051636Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4051907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4051982Z layer_outputs = layer_module( 2025-12-04T09:48:34.4052216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4052296Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4052539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4052616Z return func(*args, **kwargs) 2025-12-04T09:48:34.4052854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4052934Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4053204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4053278Z return func(*args, **kwargs) 2025-12-04T09:48:34.4053522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4053605Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4053848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4053925Z return func(*args, **kwargs) 2025-12-04T09:48:34.4054181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.4054259Z value_states = self.v(current_states) 2025-12-04T09:48:34.4054270Z 2025-12-04T09:48:34.4054350Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4054430Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4054541Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4054742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4054807Z res = mod(**inputs) 2025-12-04T09:48:34.4055057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4055131Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4055371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4055451Z layer_outputs = layer_module( 2025-12-04T09:48:34.4055675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4055770Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4056011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4056077Z return func(*args, **kwargs) 2025-12-04T09:48:34.4056320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4056397Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4056636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4056704Z return func(*args, **kwargs) 2025-12-04T09:48:34.4056933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4057023Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4057260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4057325Z return func(*args, **kwargs) 2025-12-04T09:48:34.4057564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.4057678Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.4057682Z 2025-12-04T09:48:34.4057789Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4057982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4058045Z res = mod(**inputs) 2025-12-04T09:48:34.4058283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4058356Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4058589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4058667Z layer_outputs = layer_module( 2025-12-04T09:48:34.4058884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4058986Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4059227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4059295Z return func(*args, **kwargs) 2025-12-04T09:48:34.4059530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4059610Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4059854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4059975Z return func(*args, **kwargs) 2025-12-04T09:48:34.4060205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4060299Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4060534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4060603Z return func(*args, **kwargs) 2025-12-04T09:48:34.4060843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.4060920Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.4060923Z 2025-12-04T09:48:34.4061034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4061229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4061295Z res = mod(**inputs) 2025-12-04T09:48:34.4061536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4061610Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4061841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4061924Z layer_outputs = layer_module( 2025-12-04T09:48:34.4062151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4062237Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4062477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4062547Z return func(*args, **kwargs) 2025-12-04T09:48:34.4062793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4062878Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4063128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4063197Z return func(*args, **kwargs) 2025-12-04T09:48:34.4063435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4063544Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4063800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4063872Z return func(*args, **kwargs) 2025-12-04T09:48:34.4064117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.4064193Z key_states = self.k(current_states) 2025-12-04T09:48:34.4064197Z 2025-12-04T09:48:34.4064306Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4064517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4064582Z res = mod(**inputs) 2025-12-04T09:48:34.4064823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4064914Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4065156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4065237Z layer_outputs = layer_module( 2025-12-04T09:48:34.4065462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4065549Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4065800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4065885Z return func(*args, **kwargs) 2025-12-04T09:48:34.4066123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4066203Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4066451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4066520Z return func(*args, **kwargs) 2025-12-04T09:48:34.4066751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4066841Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4067074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4067141Z return func(*args, **kwargs) 2025-12-04T09:48:34.4067377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.4067455Z value_states = self.v(current_states) 2025-12-04T09:48:34.4067458Z 2025-12-04T09:48:34.4067546Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4067626Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4067726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4067929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4067996Z res = mod(**inputs) 2025-12-04T09:48:34.4068230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4068309Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4068540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4068616Z layer_outputs = layer_module( 2025-12-04T09:48:34.4068843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4068922Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4069172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4069244Z return func(*args, **kwargs) 2025-12-04T09:48:34.4069513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4069609Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4069849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4069926Z return func(*args, **kwargs) 2025-12-04T09:48:34.4070177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4070271Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4070539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4070626Z return func(*args, **kwargs) 2025-12-04T09:48:34.4070898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.4070978Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.4070982Z 2025-12-04T09:48:34.4071087Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4071293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4071358Z res = mod(**inputs) 2025-12-04T09:48:34.4071598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4071678Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4071948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4072027Z layer_outputs = layer_module( 2025-12-04T09:48:34.4072246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4072325Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4072596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4072671Z return func(*args, **kwargs) 2025-12-04T09:48:34.4072927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4073026Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4073278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.4073412Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.4073666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.4073752Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.4073755Z 2025-12-04T09:48:34.4073872Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4074084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4074164Z res = mod(**inputs) 2025-12-04T09:48:34.4074422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4074502Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4074762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4074838Z layer_outputs = layer_module( 2025-12-04T09:48:34.4075078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4075171Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4075428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4075511Z return func(*args, **kwargs) 2025-12-04T09:48:34.4075799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4075899Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4076155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.4076281Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.4076539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.4076626Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.4076629Z 2025-12-04T09:48:34.4076740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4076955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4077023Z res = mod(**inputs) 2025-12-04T09:48:34.4077291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4077383Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4077637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4077720Z layer_outputs = layer_module( 2025-12-04T09:48:34.4077958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4078042Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4078332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4078406Z return func(*args, **kwargs) 2025-12-04T09:48:34.4078667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4078765Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4079019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.4079150Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.4079402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.4079486Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.4079490Z 2025-12-04T09:48:34.4079606Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4079819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4079899Z res = mod(**inputs) 2025-12-04T09:48:34.4080163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4080245Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4080595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4080684Z layer_outputs = layer_module( 2025-12-04T09:48:34.4080930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4081025Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4081293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4081380Z return func(*args, **kwargs) 2025-12-04T09:48:34.4081644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4081745Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4082015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 344, in forward 2025-12-04T09:48:34.4082168Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-12-04T09:48:34.4082198Z 2025-12-04T09:48:34.4082337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4082551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4082621Z res = mod(**inputs) 2025-12-04T09:48:34.4082882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4082962Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4083214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4083298Z layer_outputs = layer_module( 2025-12-04T09:48:34.4083536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4083644Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4083907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4083981Z return func(*args, **kwargs) 2025-12-04T09:48:34.4084241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4084328Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4084594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4084693Z return func(*args, **kwargs) 2025-12-04T09:48:34.4084943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4085039Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4085295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4085369Z return func(*args, **kwargs) 2025-12-04T09:48:34.4085632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.4085712Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.4085716Z 2025-12-04T09:48:34.4085827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4086025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4086090Z res = mod(**inputs) 2025-12-04T09:48:34.4086336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4086409Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4086644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4086722Z layer_outputs = layer_module( 2025-12-04T09:48:34.4086946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4087035Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4087275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4087344Z return func(*args, **kwargs) 2025-12-04T09:48:34.4087588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4087670Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4087919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4087987Z return func(*args, **kwargs) 2025-12-04T09:48:34.4088225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4088315Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4088595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4088667Z return func(*args, **kwargs) 2025-12-04T09:48:34.4088907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.4088984Z key_states = self.k(current_states) 2025-12-04T09:48:34.4088988Z 2025-12-04T09:48:34.4089097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4089298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4089363Z res = mod(**inputs) 2025-12-04T09:48:34.4089608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4089698Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4089940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4090023Z layer_outputs = layer_module( 2025-12-04T09:48:34.4090249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4090336Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4090579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4090648Z return func(*args, **kwargs) 2025-12-04T09:48:34.4090917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4090999Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4091255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4091324Z return func(*args, **kwargs) 2025-12-04T09:48:34.4091568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4091660Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4091907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4091977Z return func(*args, **kwargs) 2025-12-04T09:48:34.4092227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.4092308Z value_states = self.v(current_states) 2025-12-04T09:48:34.4092312Z 2025-12-04T09:48:34.4092402Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4092484Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4092589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4092794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4092861Z res = mod(**inputs) 2025-12-04T09:48:34.4093105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4093188Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4093430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4093510Z layer_outputs = layer_module( 2025-12-04T09:48:34.4093739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4093821Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4094072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4094140Z return func(*args, **kwargs) 2025-12-04T09:48:34.4094382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:48:34.4094523Z self_attention_outputs = self.layer[0]( 2025-12-04T09:48:34.4094767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4094844Z return func(*args, **kwargs) 2025-12-04T09:48:34.4095078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:48:34.4095160Z attention_output = self.SelfAttention( 2025-12-04T09:48:34.4095411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4095480Z return func(*args, **kwargs) 2025-12-04T09:48:34.4095724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.4095818Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.4095824Z 2025-12-04T09:48:34.4095933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4096140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4096205Z res = mod(**inputs) 2025-12-04T09:48:34.4096445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4096525Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4096765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4096869Z layer_outputs = layer_module( 2025-12-04T09:48:34.4097098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4097179Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4097431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4097503Z return func(*args, **kwargs) 2025-12-04T09:48:34.4097739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4097828Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4098069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4098144Z return func(*args, **kwargs) 2025-12-04T09:48:34.4098380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4098464Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4098709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4098779Z return func(*args, **kwargs) 2025-12-04T09:48:34.4099023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:48:34.4099101Z query_states = self.q(hidden_states) 2025-12-04T09:48:34.4099105Z 2025-12-04T09:48:34.4099207Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4099411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4099475Z res = mod(**inputs) 2025-12-04T09:48:34.4099722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4099810Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4100064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4100146Z layer_outputs = layer_module( 2025-12-04T09:48:34.4100383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4100505Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4100772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4100845Z return func(*args, **kwargs) 2025-12-04T09:48:34.4101094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4101192Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4101455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4101529Z return func(*args, **kwargs) 2025-12-04T09:48:34.4101767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4101871Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4102126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4102196Z return func(*args, **kwargs) 2025-12-04T09:48:34.4102438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:48:34.4102514Z key_states = self.k(current_states) 2025-12-04T09:48:34.4102518Z 2025-12-04T09:48:34.4102621Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4102828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4102913Z res = mod(**inputs) 2025-12-04T09:48:34.4103155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4103236Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4103479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4103562Z layer_outputs = layer_module( 2025-12-04T09:48:34.4103786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4103865Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4104114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4104183Z return func(*args, **kwargs) 2025-12-04T09:48:34.4104426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4104512Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4104756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4104834Z return func(*args, **kwargs) 2025-12-04T09:48:34.4105073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4105160Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4105411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4105481Z return func(*args, **kwargs) 2025-12-04T09:48:34.4105723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:48:34.4105805Z value_states = self.v(current_states) 2025-12-04T09:48:34.4105810Z 2025-12-04T09:48:34.4105892Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4105979Z cudagraph partition due to non gpu ops 2025-12-04T09:48:34.4106085Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4106289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4106363Z res = mod(**inputs) 2025-12-04T09:48:34.4106639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4106724Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4107016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4107089Z layer_outputs = layer_module( 2025-12-04T09:48:34.4107322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4107401Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4107640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4107717Z return func(*args, **kwargs) 2025-12-04T09:48:34.4107973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:48:34.4108071Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:48:34.4108309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4108378Z return func(*args, **kwargs) 2025-12-04T09:48:34.4108616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:48:34.4108699Z attention_output = self.EncDecAttention( 2025-12-04T09:48:34.4108942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4109032Z return func(*args, **kwargs) 2025-12-04T09:48:34.4109261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:48:34.4109342Z attn_output = self.o(attn_output) 2025-12-04T09:48:34.4109347Z 2025-12-04T09:48:34.4109450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4109646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4109719Z res = mod(**inputs) 2025-12-04T09:48:34.4109953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4110032Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4110268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4110341Z layer_outputs = layer_module( 2025-12-04T09:48:34.4110570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4110647Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4110892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4110971Z return func(*args, **kwargs) 2025-12-04T09:48:34.4111211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4111314Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4111552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.4111674Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.4111923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:48:34.4112005Z hidden_states = self.wi(hidden_states) 2025-12-04T09:48:34.4112008Z 2025-12-04T09:48:34.4112117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4112321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4112407Z res = mod(**inputs) 2025-12-04T09:48:34.4112684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4112760Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4112999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4113081Z layer_outputs = layer_module( 2025-12-04T09:48:34.4113314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4113408Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4113664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4113737Z return func(*args, **kwargs) 2025-12-04T09:48:34.4114012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4114115Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4114372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.4114496Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.4114748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:48:34.4114841Z hidden_states = self.act(hidden_states) 2025-12-04T09:48:34.4114864Z 2025-12-04T09:48:34.4114974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4115183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4115261Z res = mod(**inputs) 2025-12-04T09:48:34.4115515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:48:34.4115602Z decoder_outputs = self.decoder( 2025-12-04T09:48:34.4115857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:48:34.4115934Z layer_outputs = layer_module( 2025-12-04T09:48:34.4116177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:34.4116260Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:34.4116518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:48:34.4116599Z return func(*args, **kwargs) 2025-12-04T09:48:34.4116849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:48:34.4116951Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:48:34.4117208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:48:34.4117339Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:48:34.4117603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:48:34.4117691Z hidden_states = self.wo(hidden_states) 2025-12-04T09:48:34.4117694Z 2025-12-04T09:48:34.4117814Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4118031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4118104Z res = mod(**inputs) 2025-12-04T09:48:34.4118373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1793, in forward 2025-12-04T09:48:34.4118468Z lm_logits = self.lm_head(sequence_output) 2025-12-04T09:48:34.4118472Z 2025-12-04T09:48:34.4118587Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:34.4118862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:34.4118937Z res = mod(**inputs) 2025-12-04T09:48:34.4119209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1800, in forward 2025-12-04T09:48:34.4119369Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-12-04T09:48:34.4119373Z 2025-12-04T09:48:44.7603537Z Compilation time (from dynamo_timed): 18.180716425 2025-12-04T09:48:44.7760336Z pass 2025-12-04T09:48:44.7761063Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:44.7762299Z TIMING: _recursive_pre_grad_passes:0.01078 _recursive_joint_graph_passes:0.62731 _recursive_post_grad_passes:0.06012 async_compile.wait:0.76713 code_gen:9.88141 inductor_compile:11.18101 backend_compile:15.17161 gc:0.00023 entire_frame_compile:18.18072 total_wall_time:18.18072 2025-12-04T09:48:44.7763394Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:11485 | FakeTensor.__torch_dispatch__:4072 | ProxyTorchDispatchMode.__torch_dispatch__:3376 2025-12-04T09:48:44.7763961Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-12-04T09:48:47.2535934Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:48:47.2537199Z import pynvml # type: ignore[import] 2025-12-04T09:48:50.8914496Z 2025-12-04T09:48:51.7977860Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:48:51.7978170Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:48:51.7991885Z cpu eval T5Small 2025-12-04T09:48:53.0171362Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:53.4232055Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:53.8406647Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:49:09.1555510Z Compilation time (from dynamo_timed): 14.491081032 2025-12-04T09:49:09.1731079Z pass 2025-12-04T09:49:09.1734378Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:49:09.1735076Z TIMING: _recursive_pre_grad_passes:0.01149 _recursive_joint_graph_passes:0.60958 async_compile.wait:0.00387 inductor_compile:7.41893 backend_compile:11.47018 gc:0.00124 entire_frame_compile:14.49108 total_wall_time:14.49108 2025-12-04T09:49:09.1735874Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:9926 | FakeTensor.__torch_dispatch__:17 | ProxyTorchDispatchMode.__torch_dispatch__:3368 2025-12-04T09:49:09.1736366Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-12-04T09:49:11.4001529Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:49:11.4002450Z import pynvml # type: ignore[import] 2025-12-04T09:49:14.9571820Z 2025-12-04T09:49:17.0780827Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:49:17.0781149Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:49:17.0802420Z cpu eval TrOCRForCausalLM 2025-12-04T09:49:17.2158192Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2025-12-04T09:49:17.2599275Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:49:17.5423326Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:49:17.8153156Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:49:25.2737280Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2739480Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2739899Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2740142Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2740382Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2740602Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2740830Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2741094Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2741337Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2741549Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2741759Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2741969Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2742539Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2742967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2743319Z res = mod(**inputs) 2025-12-04T09:49:25.2743739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2744183Z outputs = self.model.decoder( 2025-12-04T09:49:25.2744602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2745021Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2745476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2745887Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2746320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2746736Z return func(*args, **kwargs) 2025-12-04T09:49:25.2747340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2747790Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2748228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2748644Z return func(*args, **kwargs) 2025-12-04T09:49:25.2749134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.2749602Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.2749789Z 2025-12-04T09:49:25.2749914Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2750368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2750725Z res = mod(**inputs) 2025-12-04T09:49:25.2751126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2751546Z outputs = self.model.decoder( 2025-12-04T09:49:25.2751970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2752393Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2752784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2753184Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2753576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2753976Z return func(*args, **kwargs) 2025-12-04T09:49:25.2754379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2754915Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2755406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2755811Z return func(*args, **kwargs) 2025-12-04T09:49:25.2756215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.2756649Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.2756800Z 2025-12-04T09:49:25.2756929Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2757323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2757676Z res = mod(**inputs) 2025-12-04T09:49:25.2758090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2758569Z outputs = self.model.decoder( 2025-12-04T09:49:25.2759029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2759464Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2759851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2760252Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2760862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2761336Z return func(*args, **kwargs) 2025-12-04T09:49:25.2761743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2762152Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2762554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2762925Z return func(*args, **kwargs) 2025-12-04T09:49:25.2763301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.2763706Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.2763853Z 2025-12-04T09:49:25.2763945Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2764159Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2764376Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2764619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2764986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2765311Z res = mod(**inputs) 2025-12-04T09:49:25.2765684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2766084Z outputs = self.model.decoder( 2025-12-04T09:49:25.2766463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2766861Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2767236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2767635Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2768014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2768394Z return func(*args, **kwargs) 2025-12-04T09:49:25.2768793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2769228Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2769644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2770100Z return func(*args, **kwargs) 2025-12-04T09:49:25.2770482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.2770880Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.2771030Z 2025-12-04T09:49:25.2771139Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2771512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2771850Z res = mod(**inputs) 2025-12-04T09:49:25.2772223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2772717Z outputs = self.model.decoder( 2025-12-04T09:49:25.2773159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2773569Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2773947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2774342Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2774754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2775146Z return func(*args, **kwargs) 2025-12-04T09:49:25.2775598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2776158Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2776364Z 2025-12-04T09:49:25.2776480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2776868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2777223Z res = mod(**inputs) 2025-12-04T09:49:25.2777617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2778040Z outputs = self.model.decoder( 2025-12-04T09:49:25.2778455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2778881Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2779265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2779662Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2780081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2780489Z return func(*args, **kwargs) 2025-12-04T09:49:25.2780892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2781374Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2781808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.2782193Z return self.act(input) 2025-12-04T09:49:25.2782310Z 2025-12-04T09:49:25.2782416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2782799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2783150Z res = mod(**inputs) 2025-12-04T09:49:25.2783541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2783964Z outputs = self.model.decoder( 2025-12-04T09:49:25.2784385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2784809Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2785239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2785632Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2786038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2786509Z return func(*args, **kwargs) 2025-12-04T09:49:25.2786904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.2787324Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.2787473Z 2025-12-04T09:49:25.2787592Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2787972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2788316Z res = mod(**inputs) 2025-12-04T09:49:25.2788722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2789144Z outputs = self.model.decoder( 2025-12-04T09:49:25.2789545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2789960Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2790336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2790716Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2791151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2791560Z return func(*args, **kwargs) 2025-12-04T09:49:25.2791967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2792407Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2792840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2793252Z return func(*args, **kwargs) 2025-12-04T09:49:25.2793624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.2794094Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.2794277Z 2025-12-04T09:49:25.2794392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2794898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2795272Z res = mod(**inputs) 2025-12-04T09:49:25.2795662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2796083Z outputs = self.model.decoder( 2025-12-04T09:49:25.2796499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2796918Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2797305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2797708Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2798117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2798528Z return func(*args, **kwargs) 2025-12-04T09:49:25.2798937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2799391Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2799834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2800276Z return func(*args, **kwargs) 2025-12-04T09:49:25.2800791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.2801249Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.2801405Z 2025-12-04T09:49:25.2801525Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2801933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2802300Z res = mod(**inputs) 2025-12-04T09:49:25.2802701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2803131Z outputs = self.model.decoder( 2025-12-04T09:49:25.2803535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2803966Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2804334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2804722Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2805120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2805526Z return func(*args, **kwargs) 2025-12-04T09:49:25.2805919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2806385Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2806797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2807180Z return func(*args, **kwargs) 2025-12-04T09:49:25.2807558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.2807976Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.2808123Z 2025-12-04T09:49:25.2808215Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2808429Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2808645Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2808883Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2809241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2809567Z res = mod(**inputs) 2025-12-04T09:49:25.2809938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2810327Z outputs = self.model.decoder( 2025-12-04T09:49:25.2810713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2811104Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2811460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2811824Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2812211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2812590Z return func(*args, **kwargs) 2025-12-04T09:49:25.2812967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2813381Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2813785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2814161Z return func(*args, **kwargs) 2025-12-04T09:49:25.2814533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.2815029Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.2815174Z 2025-12-04T09:49:25.2815278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2815640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2815956Z res = mod(**inputs) 2025-12-04T09:49:25.2816316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2816775Z outputs = self.model.decoder( 2025-12-04T09:49:25.2817149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2817534Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2817881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2818261Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2818663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2819061Z return func(*args, **kwargs) 2025-12-04T09:49:25.2819460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2819943Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2820119Z 2025-12-04T09:49:25.2820226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2820637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2820986Z res = mod(**inputs) 2025-12-04T09:49:25.2821335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2821733Z outputs = self.model.decoder( 2025-12-04T09:49:25.2822103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2822475Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2822809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2823365Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2823734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2824101Z return func(*args, **kwargs) 2025-12-04T09:49:25.2824463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2824883Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2825266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.2825600Z return self.act(input) 2025-12-04T09:49:25.2825718Z 2025-12-04T09:49:25.2825821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2826168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2826478Z res = mod(**inputs) 2025-12-04T09:49:25.2826817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2827197Z outputs = self.model.decoder( 2025-12-04T09:49:25.2827570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2827946Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2828297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2828664Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2829087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2829444Z return func(*args, **kwargs) 2025-12-04T09:49:25.2829828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.2830215Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.2830351Z 2025-12-04T09:49:25.2830459Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2830799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2831119Z res = mod(**inputs) 2025-12-04T09:49:25.2831465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2831838Z outputs = self.model.decoder( 2025-12-04T09:49:25.2832248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2832624Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2832968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2833311Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2833680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2834150Z return func(*args, **kwargs) 2025-12-04T09:49:25.2834549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2834966Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2835399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2835782Z return func(*args, **kwargs) 2025-12-04T09:49:25.2836152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.2836574Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.2836736Z 2025-12-04T09:49:25.2836846Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2837202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2837512Z res = mod(**inputs) 2025-12-04T09:49:25.2837872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2838259Z outputs = self.model.decoder( 2025-12-04T09:49:25.2838637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2839029Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2839387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2839761Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2840146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2840588Z return func(*args, **kwargs) 2025-12-04T09:49:25.2840976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2841424Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2841877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2842301Z return func(*args, **kwargs) 2025-12-04T09:49:25.2842721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.2843172Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.2843312Z 2025-12-04T09:49:25.2843437Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2843800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2844121Z res = mod(**inputs) 2025-12-04T09:49:25.2844477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2844868Z outputs = self.model.decoder( 2025-12-04T09:49:25.2845257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2845638Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2845995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2846382Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2846782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2847320Z return func(*args, **kwargs) 2025-12-04T09:49:25.2847707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2848140Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2848564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2849031Z return func(*args, **kwargs) 2025-12-04T09:49:25.2849431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.2849842Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.2849988Z 2025-12-04T09:49:25.2850076Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2850300Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2850520Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2850767Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2851126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2851453Z res = mod(**inputs) 2025-12-04T09:49:25.2851812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2852194Z outputs = self.model.decoder( 2025-12-04T09:49:25.2852568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2852951Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2853290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2853652Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2854030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2854396Z return func(*args, **kwargs) 2025-12-04T09:49:25.2854759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2855169Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2855563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2855932Z return func(*args, **kwargs) 2025-12-04T09:49:25.2856295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.2856695Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.2856829Z 2025-12-04T09:49:25.2856942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2857409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2857741Z res = mod(**inputs) 2025-12-04T09:49:25.2858112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2858513Z outputs = self.model.decoder( 2025-12-04T09:49:25.2858898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2859300Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2859690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2860065Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2860464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2860887Z return func(*args, **kwargs) 2025-12-04T09:49:25.2861287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2861729Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2861915Z 2025-12-04T09:49:25.2862029Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2862402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2862735Z res = mod(**inputs) 2025-12-04T09:49:25.2863121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2863525Z outputs = self.model.decoder( 2025-12-04T09:49:25.2863912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2864307Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2864671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2865062Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2865467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2865845Z return func(*args, **kwargs) 2025-12-04T09:49:25.2866230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2866673Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2867069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.2867433Z return self.act(input) 2025-12-04T09:49:25.2867552Z 2025-12-04T09:49:25.2867658Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2868029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2868357Z res = mod(**inputs) 2025-12-04T09:49:25.2868724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2869118Z outputs = self.model.decoder( 2025-12-04T09:49:25.2869510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2869905Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2870272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2870643Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2871024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2871408Z return func(*args, **kwargs) 2025-12-04T09:49:25.2871811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.2872233Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.2872375Z 2025-12-04T09:49:25.2872487Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2872859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2873183Z res = mod(**inputs) 2025-12-04T09:49:25.2873538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2873928Z outputs = self.model.decoder( 2025-12-04T09:49:25.2874308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2874698Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2875063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2875428Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2875808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2876183Z return func(*args, **kwargs) 2025-12-04T09:49:25.2876548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2876954Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2877374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2877745Z return func(*args, **kwargs) 2025-12-04T09:49:25.2878123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.2878561Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.2878729Z 2025-12-04T09:49:25.2878844Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2879207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2879542Z res = mod(**inputs) 2025-12-04T09:49:25.2879910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2880318Z outputs = self.model.decoder( 2025-12-04T09:49:25.2880805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2881236Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2881628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2882041Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2882434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2882826Z return func(*args, **kwargs) 2025-12-04T09:49:25.2883202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2883607Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2884015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2884398Z return func(*args, **kwargs) 2025-12-04T09:49:25.2884770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.2885177Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.2885323Z 2025-12-04T09:49:25.2885430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2885802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2886207Z res = mod(**inputs) 2025-12-04T09:49:25.2886573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2886965Z outputs = self.model.decoder( 2025-12-04T09:49:25.2887350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2887761Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2888137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2888516Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2888900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2889318Z return func(*args, **kwargs) 2025-12-04T09:49:25.2889698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2890112Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2890517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2890960Z return func(*args, **kwargs) 2025-12-04T09:49:25.2891386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.2891831Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.2891981Z 2025-12-04T09:49:25.2892074Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2892294Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2892514Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2892757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2893126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2893454Z res = mod(**inputs) 2025-12-04T09:49:25.2893825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2894225Z outputs = self.model.decoder( 2025-12-04T09:49:25.2894605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2895000Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2895374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2895744Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2896124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2896503Z return func(*args, **kwargs) 2025-12-04T09:49:25.2896889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2897305Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2897714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2898094Z return func(*args, **kwargs) 2025-12-04T09:49:25.2898480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.2898878Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.2899026Z 2025-12-04T09:49:25.2899133Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2899505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2899825Z res = mod(**inputs) 2025-12-04T09:49:25.2900176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2900605Z outputs = self.model.decoder( 2025-12-04T09:49:25.2900981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2901364Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2901720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2902091Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2902468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2902829Z return func(*args, **kwargs) 2025-12-04T09:49:25.2903199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2903726Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2903895Z 2025-12-04T09:49:25.2904003Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2904352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2904671Z res = mod(**inputs) 2025-12-04T09:49:25.2905030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2905405Z outputs = self.model.decoder( 2025-12-04T09:49:25.2905783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2906187Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2906534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2906894Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2907264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2907631Z return func(*args, **kwargs) 2025-12-04T09:49:25.2907991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2908432Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2908817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.2909152Z return self.act(input) 2025-12-04T09:49:25.2909261Z 2025-12-04T09:49:25.2909363Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2909717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2910036Z res = mod(**inputs) 2025-12-04T09:49:25.2910384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2910784Z outputs = self.model.decoder( 2025-12-04T09:49:25.2911164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2911554Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2911887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2912237Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2912603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2912972Z return func(*args, **kwargs) 2025-12-04T09:49:25.2913332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.2913721Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.2913856Z 2025-12-04T09:49:25.2913992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2914362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2914687Z res = mod(**inputs) 2025-12-04T09:49:25.2915043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2915428Z outputs = self.model.decoder( 2025-12-04T09:49:25.2915805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2916194Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2916549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2916913Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2917320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2917707Z return func(*args, **kwargs) 2025-12-04T09:49:25.2918098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2918520Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2918937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2919326Z return func(*args, **kwargs) 2025-12-04T09:49:25.2919726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.2920149Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.2920322Z 2025-12-04T09:49:25.2920496Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2920902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2921246Z res = mod(**inputs) 2025-12-04T09:49:25.2921639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2922058Z outputs = self.model.decoder( 2025-12-04T09:49:25.2922470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2922880Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2923256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2923655Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2924062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2924446Z return func(*args, **kwargs) 2025-12-04T09:49:25.2924853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2925302Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2925725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2926122Z return func(*args, **kwargs) 2025-12-04T09:49:25.2926494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.2926885Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.2927023Z 2025-12-04T09:49:25.2927129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2927488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2927810Z res = mod(**inputs) 2025-12-04T09:49:25.2928158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2928583Z outputs = self.model.decoder( 2025-12-04T09:49:25.2928992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2929389Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2929739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2930113Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2930504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2930882Z return func(*args, **kwargs) 2025-12-04T09:49:25.2931254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2931682Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2932082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2932442Z return func(*args, **kwargs) 2025-12-04T09:49:25.2932812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.2933213Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.2933354Z 2025-12-04T09:49:25.2933441Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2933649Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2933888Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2934121Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2934483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2934818Z res = mod(**inputs) 2025-12-04T09:49:25.2935195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2935590Z outputs = self.model.decoder( 2025-12-04T09:49:25.2935981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2936381Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2936738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2937111Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2937489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2937871Z return func(*args, **kwargs) 2025-12-04T09:49:25.2938255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2938667Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2939078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2939457Z return func(*args, **kwargs) 2025-12-04T09:49:25.2939829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.2940236Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.2940384Z 2025-12-04T09:49:25.2940491Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2940860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2941184Z res = mod(**inputs) 2025-12-04T09:49:25.2941549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2941946Z outputs = self.model.decoder( 2025-12-04T09:49:25.2942329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2942783Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2943142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2943513Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2943892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2944273Z return func(*args, **kwargs) 2025-12-04T09:49:25.2944659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2945100Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2945274Z 2025-12-04T09:49:25.2945382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2945769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2946109Z res = mod(**inputs) 2025-12-04T09:49:25.2946475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2946874Z outputs = self.model.decoder( 2025-12-04T09:49:25.2947408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2947815Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2948169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2948598Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2948985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2949354Z return func(*args, **kwargs) 2025-12-04T09:49:25.2949737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2950179Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2950574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.2950921Z return self.act(input) 2025-12-04T09:49:25.2951058Z 2025-12-04T09:49:25.2951162Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2951521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2951843Z res = mod(**inputs) 2025-12-04T09:49:25.2952190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2952572Z outputs = self.model.decoder( 2025-12-04T09:49:25.2952948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2953325Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2953672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2954032Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2954406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2954767Z return func(*args, **kwargs) 2025-12-04T09:49:25.2955142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.2955546Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.2955684Z 2025-12-04T09:49:25.2955790Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2956156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2956481Z res = mod(**inputs) 2025-12-04T09:49:25.2956917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2957310Z outputs = self.model.decoder( 2025-12-04T09:49:25.2957698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2958094Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2958451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2958825Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2959233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2959638Z return func(*args, **kwargs) 2025-12-04T09:49:25.2960059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2960574Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2961015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2961439Z return func(*args, **kwargs) 2025-12-04T09:49:25.2961869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.2962347Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.2962564Z 2025-12-04T09:49:25.2962696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2963053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2963377Z res = mod(**inputs) 2025-12-04T09:49:25.2963741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2964131Z outputs = self.model.decoder( 2025-12-04T09:49:25.2964505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2964886Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2965247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2965612Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2965984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2966358Z return func(*args, **kwargs) 2025-12-04T09:49:25.2966728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2967130Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2967526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2967896Z return func(*args, **kwargs) 2025-12-04T09:49:25.2968266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.2968655Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.2968797Z 2025-12-04T09:49:25.2968901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2969261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2969574Z res = mod(**inputs) 2025-12-04T09:49:25.2969937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2970322Z outputs = self.model.decoder( 2025-12-04T09:49:25.2970700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2971107Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2971473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2971825Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2972190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2972541Z return func(*args, **kwargs) 2025-12-04T09:49:25.2972901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2973298Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2973677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2974036Z return func(*args, **kwargs) 2025-12-04T09:49:25.2974417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.2974805Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.2974942Z 2025-12-04T09:49:25.2975021Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2975230Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2975434Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.2975653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2976003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2976342Z res = mod(**inputs) 2025-12-04T09:49:25.2976695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2977071Z outputs = self.model.decoder( 2025-12-04T09:49:25.2977439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2977813Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2978147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2978499Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2978864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2979227Z return func(*args, **kwargs) 2025-12-04T09:49:25.2979584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.2979983Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.2980368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2980724Z return func(*args, **kwargs) 2025-12-04T09:49:25.2981086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.2981468Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.2981600Z 2025-12-04T09:49:25.2981707Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2982046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2982357Z res = mod(**inputs) 2025-12-04T09:49:25.2982714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2983102Z outputs = self.model.decoder( 2025-12-04T09:49:25.2983467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2983848Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2984196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2984598Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2984978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2985359Z return func(*args, **kwargs) 2025-12-04T09:49:25.2985722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2986137Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2986315Z 2025-12-04T09:49:25.2986416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2986772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2987092Z res = mod(**inputs) 2025-12-04T09:49:25.2987462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2987840Z outputs = self.model.decoder( 2025-12-04T09:49:25.2988214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2988580Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2988921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2989284Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2989668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2990066Z return func(*args, **kwargs) 2025-12-04T09:49:25.2990445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.2990881Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.2991257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.2991607Z return self.act(input) 2025-12-04T09:49:25.2991725Z 2025-12-04T09:49:25.2991827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2992188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2992502Z res = mod(**inputs) 2025-12-04T09:49:25.2992857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2993245Z outputs = self.model.decoder( 2025-12-04T09:49:25.2993618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2994005Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2994357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2994722Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.2995097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.2995472Z return func(*args, **kwargs) 2025-12-04T09:49:25.2995854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.2996259Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.2996398Z 2025-12-04T09:49:25.2996505Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.2996875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.2997204Z res = mod(**inputs) 2025-12-04T09:49:25.2997580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.2998002Z outputs = self.model.decoder( 2025-12-04T09:49:25.2998479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.2998894Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.2999261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.2999650Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3000058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3000537Z return func(*args, **kwargs) 2025-12-04T09:49:25.3000978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3001450Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3001926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3002298Z return func(*args, **kwargs) 2025-12-04T09:49:25.3002674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.3003095Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.3003258Z 2025-12-04T09:49:25.3003369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3003720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3004068Z res = mod(**inputs) 2025-12-04T09:49:25.3004427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3004803Z outputs = self.model.decoder( 2025-12-04T09:49:25.3005181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3005562Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3005914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3006270Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3006650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3007022Z return func(*args, **kwargs) 2025-12-04T09:49:25.3007395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3007813Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3008215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3008593Z return func(*args, **kwargs) 2025-12-04T09:49:25.3008976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.3009392Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.3009529Z 2025-12-04T09:49:25.3009643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3010011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3010333Z res = mod(**inputs) 2025-12-04T09:49:25.3010699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3011095Z outputs = self.model.decoder( 2025-12-04T09:49:25.3011483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3011876Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3012230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3012638Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3013046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3013426Z return func(*args, **kwargs) 2025-12-04T09:49:25.3013809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3014217Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3014624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3015002Z return func(*args, **kwargs) 2025-12-04T09:49:25.3015382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.3015785Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.3015966Z 2025-12-04T09:49:25.3016054Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3016283Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3016492Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3016736Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3017100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3017433Z res = mod(**inputs) 2025-12-04T09:49:25.3017797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3018219Z outputs = self.model.decoder( 2025-12-04T09:49:25.3018610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3019004Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3019357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3019735Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3020123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3020494Z return func(*args, **kwargs) 2025-12-04T09:49:25.3020874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3021297Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3021683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3022036Z return func(*args, **kwargs) 2025-12-04T09:49:25.3022389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.3022771Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.3022903Z 2025-12-04T09:49:25.3023006Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3023357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3023665Z res = mod(**inputs) 2025-12-04T09:49:25.3024012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3024380Z outputs = self.model.decoder( 2025-12-04T09:49:25.3024744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3025122Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3025451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3025803Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3026171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3026563Z return func(*args, **kwargs) 2025-12-04T09:49:25.3026934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3027354Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3027528Z 2025-12-04T09:49:25.3027630Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3027978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3028286Z res = mod(**inputs) 2025-12-04T09:49:25.3028636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3029009Z outputs = self.model.decoder( 2025-12-04T09:49:25.3029390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3029770Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3030110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3030463Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3030825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3031188Z return func(*args, **kwargs) 2025-12-04T09:49:25.3031550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3031989Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3032375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.3032709Z return self.act(input) 2025-12-04T09:49:25.3032817Z 2025-12-04T09:49:25.3032935Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3033287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3033610Z res = mod(**inputs) 2025-12-04T09:49:25.3033967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3034347Z outputs = self.model.decoder( 2025-12-04T09:49:25.3034713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3035096Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3035439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3035790Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3036167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3036537Z return func(*args, **kwargs) 2025-12-04T09:49:25.3036916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.3037308Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.3037462Z 2025-12-04T09:49:25.3037575Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3037960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3038299Z res = mod(**inputs) 2025-12-04T09:49:25.3038683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3039096Z outputs = self.model.decoder( 2025-12-04T09:49:25.3039495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3039899Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3040324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3040827Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3041248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3041665Z return func(*args, **kwargs) 2025-12-04T09:49:25.3042073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3042502Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3042899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3043286Z return func(*args, **kwargs) 2025-12-04T09:49:25.3043699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.3044136Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.3044302Z 2025-12-04T09:49:25.3044407Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3044768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3045092Z res = mod(**inputs) 2025-12-04T09:49:25.3045448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3045855Z outputs = self.model.decoder( 2025-12-04T09:49:25.3046236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3046615Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3046961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3047428Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3047815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3048188Z return func(*args, **kwargs) 2025-12-04T09:49:25.3048556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3048966Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3049370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3049736Z return func(*args, **kwargs) 2025-12-04T09:49:25.3050110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.3050516Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.3050650Z 2025-12-04T09:49:25.3050763Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3051114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3051434Z res = mod(**inputs) 2025-12-04T09:49:25.3051791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3052170Z outputs = self.model.decoder( 2025-12-04T09:49:25.3052551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3052925Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3053263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3053611Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3053977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3054381Z return func(*args, **kwargs) 2025-12-04T09:49:25.3054772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3055163Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3055549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3055909Z return func(*args, **kwargs) 2025-12-04T09:49:25.3056266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.3056656Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.3056804Z 2025-12-04T09:49:25.3056890Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3057101Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3057301Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3057566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3057921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3058224Z res = mod(**inputs) 2025-12-04T09:49:25.3058581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3059010Z outputs = self.model.decoder( 2025-12-04T09:49:25.3059378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3059770Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3060108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3060465Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3060831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3061200Z return func(*args, **kwargs) 2025-12-04T09:49:25.3061568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3061971Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3062353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3062719Z return func(*args, **kwargs) 2025-12-04T09:49:25.3063088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.3063483Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.3063618Z 2025-12-04T09:49:25.3063722Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3064079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3064396Z res = mod(**inputs) 2025-12-04T09:49:25.3064745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3065128Z outputs = self.model.decoder( 2025-12-04T09:49:25.3065501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3065878Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3066215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3066579Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3066944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3067297Z return func(*args, **kwargs) 2025-12-04T09:49:25.3067673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3068130Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3068299Z 2025-12-04T09:49:25.3068407Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3068999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3069314Z res = mod(**inputs) 2025-12-04T09:49:25.3069667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3070048Z outputs = self.model.decoder( 2025-12-04T09:49:25.3070438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3070840Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3071236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3071619Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3072032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3072426Z return func(*args, **kwargs) 2025-12-04T09:49:25.3072815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3073232Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3073619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.3073979Z return self.act(input) 2025-12-04T09:49:25.3074090Z 2025-12-04T09:49:25.3074195Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3074558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3074871Z res = mod(**inputs) 2025-12-04T09:49:25.3075224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3075588Z outputs = self.model.decoder( 2025-12-04T09:49:25.3075953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3076325Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3076664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3077024Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3077400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3077769Z return func(*args, **kwargs) 2025-12-04T09:49:25.3078135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.3078540Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.3078680Z 2025-12-04T09:49:25.3078796Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3079165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3079485Z res = mod(**inputs) 2025-12-04T09:49:25.3079852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3080249Z outputs = self.model.decoder( 2025-12-04T09:49:25.3080708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3081146Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3081536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3081938Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3082359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3082723Z return func(*args, **kwargs) 2025-12-04T09:49:25.3083083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3083477Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3083871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3084237Z return func(*args, **kwargs) 2025-12-04T09:49:25.3084605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.3085024Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.3085199Z 2025-12-04T09:49:25.3085321Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3085683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3086007Z res = mod(**inputs) 2025-12-04T09:49:25.3086354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3086739Z outputs = self.model.decoder( 2025-12-04T09:49:25.3087115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3087514Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3087889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3088252Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3088627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3088987Z return func(*args, **kwargs) 2025-12-04T09:49:25.3089362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3089785Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3090181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3090557Z return func(*args, **kwargs) 2025-12-04T09:49:25.3090935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.3091342Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.3091476Z 2025-12-04T09:49:25.3091579Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3091934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3092249Z res = mod(**inputs) 2025-12-04T09:49:25.3092605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3092980Z outputs = self.model.decoder( 2025-12-04T09:49:25.3093351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3093728Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3094067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3094426Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3094796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3095161Z return func(*args, **kwargs) 2025-12-04T09:49:25.3095524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3095930Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3096383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3096754Z return func(*args, **kwargs) 2025-12-04T09:49:25.3097122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.3097522Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.3097670Z 2025-12-04T09:49:25.3097759Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3097973Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3098187Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3098423Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3098775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3099117Z res = mod(**inputs) 2025-12-04T09:49:25.3099491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3099882Z outputs = self.model.decoder( 2025-12-04T09:49:25.3100263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3100660Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3101020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3101412Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3101804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3102174Z return func(*args, **kwargs) 2025-12-04T09:49:25.3102544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3102945Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3103341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3103713Z return func(*args, **kwargs) 2025-12-04T09:49:25.3104090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.3104487Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.3104631Z 2025-12-04T09:49:25.3104736Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3105105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3105426Z res = mod(**inputs) 2025-12-04T09:49:25.3105790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3106188Z outputs = self.model.decoder( 2025-12-04T09:49:25.3106574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3106959Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3107313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3107685Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3108069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3108445Z return func(*args, **kwargs) 2025-12-04T09:49:25.3108822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3109262Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3109441Z 2025-12-04T09:49:25.3109550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3109967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3110303Z res = mod(**inputs) 2025-12-04T09:49:25.3110679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3111078Z outputs = self.model.decoder( 2025-12-04T09:49:25.3111475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3111879Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3112241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3112620Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3113037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3113426Z return func(*args, **kwargs) 2025-12-04T09:49:25.3113784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3114202Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3114582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.3114922Z return self.act(input) 2025-12-04T09:49:25.3115030Z 2025-12-04T09:49:25.3115131Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3115505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3115821Z res = mod(**inputs) 2025-12-04T09:49:25.3116166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3116549Z outputs = self.model.decoder( 2025-12-04T09:49:25.3116934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3117317Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3117655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3118013Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3118388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3118756Z return func(*args, **kwargs) 2025-12-04T09:49:25.3119141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.3119546Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.3119687Z 2025-12-04T09:49:25.3119799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3120179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3120609Z res = mod(**inputs) 2025-12-04T09:49:25.3121002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3121415Z outputs = self.model.decoder( 2025-12-04T09:49:25.3121835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3122230Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3122589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3122956Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3123338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3123721Z return func(*args, **kwargs) 2025-12-04T09:49:25.3124137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3124563Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3124961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3125334Z return func(*args, **kwargs) 2025-12-04T09:49:25.3125695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.3126126Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.3126296Z 2025-12-04T09:49:25.3126400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3126762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3127077Z res = mod(**inputs) 2025-12-04T09:49:25.3127458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3127851Z outputs = self.model.decoder( 2025-12-04T09:49:25.3128223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3128609Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3128956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3129315Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3129705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3130073Z return func(*args, **kwargs) 2025-12-04T09:49:25.3130442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3130852Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3131244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3131616Z return func(*args, **kwargs) 2025-12-04T09:49:25.3131988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.3132381Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.3132528Z 2025-12-04T09:49:25.3132635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3133004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3133337Z res = mod(**inputs) 2025-12-04T09:49:25.3133699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3134100Z outputs = self.model.decoder( 2025-12-04T09:49:25.3134481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3134872Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3135224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3135595Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3135982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3136355Z return func(*args, **kwargs) 2025-12-04T09:49:25.3136739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3137147Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3137545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3137936Z return func(*args, **kwargs) 2025-12-04T09:49:25.3138321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.3138718Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.3138859Z 2025-12-04T09:49:25.3138943Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3139161Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3139376Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3139613Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3139968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3140294Z res = mod(**inputs) 2025-12-04T09:49:25.3140658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3141090Z outputs = self.model.decoder( 2025-12-04T09:49:25.3141488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3141872Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3142236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3142595Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3142980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3143392Z return func(*args, **kwargs) 2025-12-04T09:49:25.3143753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3144166Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3144562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3144930Z return func(*args, **kwargs) 2025-12-04T09:49:25.3145292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.3145683Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.3145818Z 2025-12-04T09:49:25.3145931Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3146288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3146604Z res = mod(**inputs) 2025-12-04T09:49:25.3146966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3147476Z outputs = self.model.decoder( 2025-12-04T09:49:25.3147849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3148241Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3148596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3148956Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3149435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3149811Z return func(*args, **kwargs) 2025-12-04T09:49:25.3150193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3150629Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3150814Z 2025-12-04T09:49:25.3150921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3151295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3151619Z res = mod(**inputs) 2025-12-04T09:49:25.3151970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3152471Z outputs = self.model.decoder( 2025-12-04T09:49:25.3152864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3153262Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3153612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3153981Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3154375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3154750Z return func(*args, **kwargs) 2025-12-04T09:49:25.3155135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3155606Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3156010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.3156356Z return self.act(input) 2025-12-04T09:49:25.3156477Z 2025-12-04T09:49:25.3156585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3156954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3157277Z res = mod(**inputs) 2025-12-04T09:49:25.3157647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3158073Z outputs = self.model.decoder( 2025-12-04T09:49:25.3158465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3158855Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3159215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3159594Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3159981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3160364Z return func(*args, **kwargs) 2025-12-04T09:49:25.3160813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.3161248Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.3161402Z 2025-12-04T09:49:25.3161514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3161901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3162248Z res = mod(**inputs) 2025-12-04T09:49:25.3162640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3163053Z outputs = self.model.decoder( 2025-12-04T09:49:25.3163443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3163839Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3164188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3164565Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3164951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3165333Z return func(*args, **kwargs) 2025-12-04T09:49:25.3165713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3166138Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3166570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3166971Z return func(*args, **kwargs) 2025-12-04T09:49:25.3167342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.3167769Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.3167926Z 2025-12-04T09:49:25.3168037Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3168380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3168703Z res = mod(**inputs) 2025-12-04T09:49:25.3169062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3169453Z outputs = self.model.decoder( 2025-12-04T09:49:25.3169843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3170233Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3170584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3170944Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3171375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3171748Z return func(*args, **kwargs) 2025-12-04T09:49:25.3172154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3172547Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3172935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3173299Z return func(*args, **kwargs) 2025-12-04T09:49:25.3173665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.3174043Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.3174181Z 2025-12-04T09:49:25.3174282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3174632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3174941Z res = mod(**inputs) 2025-12-04T09:49:25.3175290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3175670Z outputs = self.model.decoder( 2025-12-04T09:49:25.3176037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3176403Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3176747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3177100Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3177454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3177821Z return func(*args, **kwargs) 2025-12-04T09:49:25.3178180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3178575Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3178956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3179318Z return func(*args, **kwargs) 2025-12-04T09:49:25.3179677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.3180064Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.3180225Z 2025-12-04T09:49:25.3180323Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3180533Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3180738Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3180962Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3181317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3181634Z res = mod(**inputs) 2025-12-04T09:49:25.3181982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3182368Z outputs = self.model.decoder( 2025-12-04T09:49:25.3182743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3183128Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3183487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3183848Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3184227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3184599Z return func(*args, **kwargs) 2025-12-04T09:49:25.3184967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3185382Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3185799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3186222Z return func(*args, **kwargs) 2025-12-04T09:49:25.3186587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.3186967Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.3187099Z 2025-12-04T09:49:25.3187206Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3187545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3187856Z res = mod(**inputs) 2025-12-04T09:49:25.3188209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3188596Z outputs = self.model.decoder( 2025-12-04T09:49:25.3188961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3189335Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3189671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3190016Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3190380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3190739Z return func(*args, **kwargs) 2025-12-04T09:49:25.3191096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3191506Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3191680Z 2025-12-04T09:49:25.3191780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3192128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3192434Z res = mod(**inputs) 2025-12-04T09:49:25.3192785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3193156Z outputs = self.model.decoder( 2025-12-04T09:49:25.3193520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3193936Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3194276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3194635Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3195005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3195373Z return func(*args, **kwargs) 2025-12-04T09:49:25.3195743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3196172Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3196559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.3196929Z return self.act(input) 2025-12-04T09:49:25.3197046Z 2025-12-04T09:49:25.3197160Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3197528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3197851Z res = mod(**inputs) 2025-12-04T09:49:25.3198220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3198613Z outputs = self.model.decoder( 2025-12-04T09:49:25.3198990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3199433Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3199814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3200210Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3200687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3201104Z return func(*args, **kwargs) 2025-12-04T09:49:25.3201512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.3201936Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.3202098Z 2025-12-04T09:49:25.3202214Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3202612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3202944Z res = mod(**inputs) 2025-12-04T09:49:25.3203308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3203709Z outputs = self.model.decoder( 2025-12-04T09:49:25.3204096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3204490Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3204844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3205214Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3205599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3205973Z return func(*args, **kwargs) 2025-12-04T09:49:25.3206352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3206773Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3207183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3207555Z return func(*args, **kwargs) 2025-12-04T09:49:25.3207964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:49:25.3208447Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:49:25.3208624Z 2025-12-04T09:49:25.3208735Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3209123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3209469Z res = mod(**inputs) 2025-12-04T09:49:25.3209860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3210248Z outputs = self.model.decoder( 2025-12-04T09:49:25.3210636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3211027Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3211404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3211769Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3212147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3212524Z return func(*args, **kwargs) 2025-12-04T09:49:25.3212894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3213317Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3213745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3214136Z return func(*args, **kwargs) 2025-12-04T09:49:25.3214512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:49:25.3214915Z key_states = self.k_proj(current_states) 2025-12-04T09:49:25.3215056Z 2025-12-04T09:49:25.3215169Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3215535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3215856Z res = mod(**inputs) 2025-12-04T09:49:25.3216220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3216637Z outputs = self.model.decoder( 2025-12-04T09:49:25.3217015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3217411Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3217769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3218137Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3218521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3218908Z return func(*args, **kwargs) 2025-12-04T09:49:25.3219291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3219702Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3220114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3220492Z return func(*args, **kwargs) 2025-12-04T09:49:25.3220884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:49:25.3221361Z value_states = self.v_proj(current_states) 2025-12-04T09:49:25.3221514Z 2025-12-04T09:49:25.3221597Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3221823Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3222032Z cudagraph partition due to non gpu ops 2025-12-04T09:49:25.3222327Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3222696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3223024Z res = mod(**inputs) 2025-12-04T09:49:25.3223388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3223788Z outputs = self.model.decoder( 2025-12-04T09:49:25.3224177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3224565Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3224920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3225288Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3225690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3226063Z return func(*args, **kwargs) 2025-12-04T09:49:25.3226447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:49:25.3226870Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:49:25.3227268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3227650Z return func(*args, **kwargs) 2025-12-04T09:49:25.3228062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:49:25.3228470Z attn_output = self.out_proj(attn_output) 2025-12-04T09:49:25.3228611Z 2025-12-04T09:49:25.3228717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3229091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3229427Z res = mod(**inputs) 2025-12-04T09:49:25.3229788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3230185Z outputs = self.model.decoder( 2025-12-04T09:49:25.3230581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3230980Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3231329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3231707Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3232091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3232468Z return func(*args, **kwargs) 2025-12-04T09:49:25.3232846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3233295Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3233473Z 2025-12-04T09:49:25.3233585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3233945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3234279Z res = mod(**inputs) 2025-12-04T09:49:25.3234639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3235030Z outputs = self.model.decoder( 2025-12-04T09:49:25.3235403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3235793Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3236156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3236578Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3236978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3237384Z return func(*args, **kwargs) 2025-12-04T09:49:25.3237790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:49:25.3238247Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:49:25.3238674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:49:25.3239044Z return self.act(input) 2025-12-04T09:49:25.3239164Z 2025-12-04T09:49:25.3239284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3239684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3240039Z res = mod(**inputs) 2025-12-04T09:49:25.3240505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:49:25.3240932Z outputs = self.model.decoder( 2025-12-04T09:49:25.3241348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:49:25.3241775Z layer_outputs = decoder_layer( 2025-12-04T09:49:25.3242166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:49:25.3242588Z return super().__call__(*args, **kwargs) 2025-12-04T09:49:25.3242998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:49:25.3243401Z return func(*args, **kwargs) 2025-12-04T09:49:25.3243798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:49:25.3244204Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:49:25.3244361Z 2025-12-04T09:49:25.3244474Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3244862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3245200Z res = mod(**inputs) 2025-12-04T09:49:25.3245591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 844, in forward 2025-12-04T09:49:25.3246029Z logits = self.output_projection(outputs[0]) 2025-12-04T09:49:25.3246188Z 2025-12-04T09:49:25.3246309Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:49:25.3246689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:49:25.3247037Z res = mod(**inputs) 2025-12-04T09:49:25.3247545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 849, in forward 2025-12-04T09:49:25.3248040Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:49:25.3248259Z 2025-12-04T09:49:35.7282044Z Compilation time (from dynamo_timed): 17.201229358 2025-12-04T09:49:35.7326601Z pass 2025-12-04T09:49:35.7327049Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:49:35.7327930Z TIMING: _recursive_pre_grad_passes:0.00806 _recursive_joint_graph_passes:0.72147 _recursive_post_grad_passes:0.06642 async_compile.wait:1.00429 code_gen:9.93639 inductor_compile:11.21404 backend_compile:14.58105 gc:0.00023 entire_frame_compile:17.20123 total_wall_time:17.20123 2025-12-04T09:49:35.7329003Z STATS: call_* op count: 443 | FakeTensorMode.__torch_dispatch__:8334 | FakeTensor.__torch_dispatch__:4316 | ProxyTorchDispatchMode.__torch_dispatch__:2529 2025-12-04T09:49:35.7329577Z Dynamo produced 1 graphs covering 443 ops with 0 graph breaks (0 unique) 2025-12-04T09:49:38.4708756Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:49:38.4711153Z import pynvml # type: ignore[import] 2025-12-04T09:49:41.9792464Z 2025-12-04T09:49:47.5034632Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:49:47.5035025Z loading model: 0it [00:05, ?it/s] 2025-12-04T09:49:47.5061290Z cpu eval XGLMForCausalLM 2025-12-04T09:49:47.9125540Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2025-12-04T09:49:47.9999534Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:49:48.5043538Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:49:49.0125207Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:50:03.5921328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5921890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5922264Z res = mod(**inputs) 2025-12-04T09:50:03.5922685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5923508Z outputs = self.model( 2025-12-04T09:50:03.5923927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5924368Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5924774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5925182Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5925627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5926041Z return func(*args, **kwargs) 2025-12-04T09:50:03.5926457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.5926929Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.5927402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5927824Z return func(*args, **kwargs) 2025-12-04T09:50:03.5928245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.5928749Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.5928945Z 2025-12-04T09:50:03.5929079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5929489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5929891Z res = mod(**inputs) 2025-12-04T09:50:03.5930282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5930700Z outputs = self.model( 2025-12-04T09:50:03.5931104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5931546Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5932134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5932539Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5932980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5933490Z return func(*args, **kwargs) 2025-12-04T09:50:03.5934073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.5934548Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.5935006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5935423Z return func(*args, **kwargs) 2025-12-04T09:50:03.5935840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.5936301Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.5936467Z 2025-12-04T09:50:03.5936587Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5937005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5937400Z res = mod(**inputs) 2025-12-04T09:50:03.5937803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5938229Z outputs = self.model( 2025-12-04T09:50:03.5938679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5939108Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5939504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5939966Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5940390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5940802Z return func(*args, **kwargs) 2025-12-04T09:50:03.5941211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.5941663Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.5942110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5942520Z return func(*args, **kwargs) 2025-12-04T09:50:03.5942931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.5943389Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.5943575Z 2025-12-04T09:50:03.5943695Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5944092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5944447Z res = mod(**inputs) 2025-12-04T09:50:03.5944834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5945263Z outputs = self.model( 2025-12-04T09:50:03.5945674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5946090Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5946480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5946948Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5947628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5948038Z return func(*args, **kwargs) 2025-12-04T09:50:03.5948450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.5948903Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.5949350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5949838Z return func(*args, **kwargs) 2025-12-04T09:50:03.5950246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.5950743Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.5950958Z 2025-12-04T09:50:03.5951077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5951483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5951860Z res = mod(**inputs) 2025-12-04T09:50:03.5952262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5952692Z outputs = self.model( 2025-12-04T09:50:03.5953150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5953587Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5953978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5954371Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5954779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5955246Z return func(*args, **kwargs) 2025-12-04T09:50:03.5955648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.5956129Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.5956573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5956990Z return func(*args, **kwargs) 2025-12-04T09:50:03.5957397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.5957841Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.5957998Z 2025-12-04T09:50:03.5958123Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5958518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5958914Z res = mod(**inputs) 2025-12-04T09:50:03.5959315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5959738Z outputs = self.model( 2025-12-04T09:50:03.5960128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5960681Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5961086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5961491Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5961910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5962311Z return func(*args, **kwargs) 2025-12-04T09:50:03.5962709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.5963255Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.5963697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5964105Z return func(*args, **kwargs) 2025-12-04T09:50:03.5964511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.5964947Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.5965123Z 2025-12-04T09:50:03.5965271Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5965686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5966040Z res = mod(**inputs) 2025-12-04T09:50:03.5966436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5966840Z outputs = self.model( 2025-12-04T09:50:03.5967225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5967631Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5968011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5968404Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5968820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5969226Z return func(*args, **kwargs) 2025-12-04T09:50:03.5969634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.5970073Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.5970494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5970899Z return func(*args, **kwargs) 2025-12-04T09:50:03.5971296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.5971791Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.5971987Z 2025-12-04T09:50:03.5972102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5972492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5972856Z res = mod(**inputs) 2025-12-04T09:50:03.5973257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5973666Z outputs = self.model( 2025-12-04T09:50:03.5974057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5974459Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5974836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5975232Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5975639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5976050Z return func(*args, **kwargs) 2025-12-04T09:50:03.5976458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.5976911Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.5977349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5977763Z return func(*args, **kwargs) 2025-12-04T09:50:03.5978158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.5978574Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.5978728Z 2025-12-04T09:50:03.5978839Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5979222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5979568Z res = mod(**inputs) 2025-12-04T09:50:03.5979942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5980368Z outputs = self.model( 2025-12-04T09:50:03.5980784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5981226Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5981602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5981997Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5982409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5982809Z return func(*args, **kwargs) 2025-12-04T09:50:03.5983209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.5983684Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.5983872Z 2025-12-04T09:50:03.5984013Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5984402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5984756Z res = mod(**inputs) 2025-12-04T09:50:03.5985157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5985569Z outputs = self.model( 2025-12-04T09:50:03.5985970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5986426Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5986806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5987195Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5987618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5988034Z return func(*args, **kwargs) 2025-12-04T09:50:03.5988443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.5988911Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.5989344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.5989734Z return self.act(input) 2025-12-04T09:50:03.5989857Z 2025-12-04T09:50:03.5989970Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5990370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5990725Z res = mod(**inputs) 2025-12-04T09:50:03.5991119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5991616Z outputs = self.model( 2025-12-04T09:50:03.5992014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5992438Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5992819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5993219Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5993639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5994053Z return func(*args, **kwargs) 2025-12-04T09:50:03.5994452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.5994899Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.5995051Z 2025-12-04T09:50:03.5995173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.5995570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.5995961Z res = mod(**inputs) 2025-12-04T09:50:03.5996360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.5996802Z outputs = self.model( 2025-12-04T09:50:03.5997197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.5997632Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.5998032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.5998443Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.5998867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.5999325Z return func(*args, **kwargs) 2025-12-04T09:50:03.5999746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6000194Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6000747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6001176Z return func(*args, **kwargs) 2025-12-04T09:50:03.6001590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6002164Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6002355Z 2025-12-04T09:50:03.6002474Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6002888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6003235Z res = mod(**inputs) 2025-12-04T09:50:03.6003612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6004024Z outputs = self.model( 2025-12-04T09:50:03.6004410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6004828Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6005220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6005626Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6006035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6006429Z return func(*args, **kwargs) 2025-12-04T09:50:03.6006825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6007278Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6007713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6008117Z return func(*args, **kwargs) 2025-12-04T09:50:03.6008512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6008933Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6009078Z 2025-12-04T09:50:03.6009189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6009576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6009923Z res = mod(**inputs) 2025-12-04T09:50:03.6010303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6010701Z outputs = self.model( 2025-12-04T09:50:03.6011088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6011537Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6011912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6012316Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6012740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6013151Z return func(*args, **kwargs) 2025-12-04T09:50:03.6013543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6013981Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6014413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6014835Z return func(*args, **kwargs) 2025-12-04T09:50:03.6015239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6015692Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6015866Z 2025-12-04T09:50:03.6015987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6016368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6016744Z res = mod(**inputs) 2025-12-04T09:50:03.6017144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6017601Z outputs = self.model( 2025-12-04T09:50:03.6017981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6018391Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6018769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6019159Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6019566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6019961Z return func(*args, **kwargs) 2025-12-04T09:50:03.6020355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6020791Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6021223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6021628Z return func(*args, **kwargs) 2025-12-04T09:50:03.6022033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6022504Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6022716Z 2025-12-04T09:50:03.6022830Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6023214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6023556Z res = mod(**inputs) 2025-12-04T09:50:03.6023937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6024342Z outputs = self.model( 2025-12-04T09:50:03.6024725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6025129Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6025525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6025922Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6026355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6026758Z return func(*args, **kwargs) 2025-12-04T09:50:03.6027158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6027596Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6028017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6028416Z return func(*args, **kwargs) 2025-12-04T09:50:03.6028813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6029253Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6029406Z 2025-12-04T09:50:03.6029535Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6029925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6030275Z res = mod(**inputs) 2025-12-04T09:50:03.6030679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6031095Z outputs = self.model( 2025-12-04T09:50:03.6031473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6031883Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6032288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6032682Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6033094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6033575Z return func(*args, **kwargs) 2025-12-04T09:50:03.6033982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6034411Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6034850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6035263Z return func(*args, **kwargs) 2025-12-04T09:50:03.6035651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6036094Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6036265Z 2025-12-04T09:50:03.6036378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6036760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6037099Z res = mod(**inputs) 2025-12-04T09:50:03.6037481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6037895Z outputs = self.model( 2025-12-04T09:50:03.6038293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6038720Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6039095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6039493Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6039903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6040308Z return func(*args, **kwargs) 2025-12-04T09:50:03.6040791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6041262Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6041748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6042173Z return func(*args, **kwargs) 2025-12-04T09:50:03.6042581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6043050Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6043231Z 2025-12-04T09:50:03.6043337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6043700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6044030Z res = mod(**inputs) 2025-12-04T09:50:03.6044381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6044782Z outputs = self.model( 2025-12-04T09:50:03.6045155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6045556Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6045913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6046291Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6046682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6047248Z return func(*args, **kwargs) 2025-12-04T09:50:03.6047640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6048062Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6048479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6048857Z return func(*args, **kwargs) 2025-12-04T09:50:03.6049241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6049646Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6049799Z 2025-12-04T09:50:03.6049912Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6050266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6050587Z res = mod(**inputs) 2025-12-04T09:50:03.6050945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6051314Z outputs = self.model( 2025-12-04T09:50:03.6051676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6052060Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6052414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6052770Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6053149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6053522Z return func(*args, **kwargs) 2025-12-04T09:50:03.6053879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6054310Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6054490Z 2025-12-04T09:50:03.6054595Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6054951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6055260Z res = mod(**inputs) 2025-12-04T09:50:03.6055611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6056076Z outputs = self.model( 2025-12-04T09:50:03.6056422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6056802Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6057145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6057504Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6057884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6058264Z return func(*args, **kwargs) 2025-12-04T09:50:03.6058637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6059098Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6059504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6059847Z return self.act(input) 2025-12-04T09:50:03.6059957Z 2025-12-04T09:50:03.6060070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6060420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6060746Z res = mod(**inputs) 2025-12-04T09:50:03.6061108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6061548Z outputs = self.model( 2025-12-04T09:50:03.6061934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6062328Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6062690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6063044Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6063417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6063784Z return func(*args, **kwargs) 2025-12-04T09:50:03.6064148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6064528Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6064672Z 2025-12-04T09:50:03.6064777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6065132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6065452Z res = mod(**inputs) 2025-12-04T09:50:03.6065804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6066186Z outputs = self.model( 2025-12-04T09:50:03.6066551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6066935Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6067280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6067636Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6068006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6068367Z return func(*args, **kwargs) 2025-12-04T09:50:03.6068737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6069152Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6069551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6069963Z return func(*args, **kwargs) 2025-12-04T09:50:03.6070339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6070769Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6070936Z 2025-12-04T09:50:03.6071041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6071404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6071732Z res = mod(**inputs) 2025-12-04T09:50:03.6072091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6072468Z outputs = self.model( 2025-12-04T09:50:03.6072847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6073237Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6073586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6073957Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6074343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6074723Z return func(*args, **kwargs) 2025-12-04T09:50:03.6075087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6075524Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6075934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6076309Z return func(*args, **kwargs) 2025-12-04T09:50:03.6076686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6077088Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6077226Z 2025-12-04T09:50:03.6077339Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6077697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6078027Z res = mod(**inputs) 2025-12-04T09:50:03.6078388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6078776Z outputs = self.model( 2025-12-04T09:50:03.6079137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6079529Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6079901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6080291Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6080780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6081198Z return func(*args, **kwargs) 2025-12-04T09:50:03.6081605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6082018Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6082434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6082820Z return func(*args, **kwargs) 2025-12-04T09:50:03.6083188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6083622Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6083800Z 2025-12-04T09:50:03.6083933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6084316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6084640Z res = mod(**inputs) 2025-12-04T09:50:03.6085005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6085396Z outputs = self.model( 2025-12-04T09:50:03.6085757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6086157Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6086509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6086876Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6087264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6087640Z return func(*args, **kwargs) 2025-12-04T09:50:03.6088009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6088415Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6088815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6089189Z return func(*args, **kwargs) 2025-12-04T09:50:03.6089563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6090029Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6090229Z 2025-12-04T09:50:03.6090336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6090707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6091026Z res = mod(**inputs) 2025-12-04T09:50:03.6091370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6091752Z outputs = self.model( 2025-12-04T09:50:03.6092113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6092498Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6092847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6093216Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6093599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6093974Z return func(*args, **kwargs) 2025-12-04T09:50:03.6094338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6094747Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6095139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6095502Z return func(*args, **kwargs) 2025-12-04T09:50:03.6095868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6096263Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6096406Z 2025-12-04T09:50:03.6096511Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6096867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6097187Z res = mod(**inputs) 2025-12-04T09:50:03.6097542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6097927Z outputs = self.model( 2025-12-04T09:50:03.6098295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6098682Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6099030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6099401Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6099783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6100162Z return func(*args, **kwargs) 2025-12-04T09:50:03.6100532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6101124Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6101552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6101931Z return func(*args, **kwargs) 2025-12-04T09:50:03.6102300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6102705Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6102858Z 2025-12-04T09:50:03.6102969Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6103320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6103662Z res = mod(**inputs) 2025-12-04T09:50:03.6104011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6104385Z outputs = self.model( 2025-12-04T09:50:03.6104735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6105116Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6105465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6105818Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6106198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6106563Z return func(*args, **kwargs) 2025-12-04T09:50:03.6106925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6107321Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6107713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6108079Z return func(*args, **kwargs) 2025-12-04T09:50:03.6108440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6108864Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6109046Z 2025-12-04T09:50:03.6109148Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6109499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6109816Z res = mod(**inputs) 2025-12-04T09:50:03.6110158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6110533Z outputs = self.model( 2025-12-04T09:50:03.6110889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6111263Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6111610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6112015Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6112388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6112784Z return func(*args, **kwargs) 2025-12-04T09:50:03.6113141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6113537Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6113920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6114287Z return func(*args, **kwargs) 2025-12-04T09:50:03.6114650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6115059Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6115201Z 2025-12-04T09:50:03.6115306Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6115665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6115998Z res = mod(**inputs) 2025-12-04T09:50:03.6116354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6116745Z outputs = self.model( 2025-12-04T09:50:03.6117139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6117582Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6117955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6118357Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6118777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6119184Z return func(*args, **kwargs) 2025-12-04T09:50:03.6119592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6120063Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6120250Z 2025-12-04T09:50:03.6120371Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6120832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6121215Z res = mod(**inputs) 2025-12-04T09:50:03.6121629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6122071Z outputs = self.model( 2025-12-04T09:50:03.6122457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6122865Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6123233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6123717Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6124114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6124504Z return func(*args, **kwargs) 2025-12-04T09:50:03.6124888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6125325Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6125733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6126092Z return self.act(input) 2025-12-04T09:50:03.6126207Z 2025-12-04T09:50:03.6126317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6126789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6127118Z res = mod(**inputs) 2025-12-04T09:50:03.6127485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6127870Z outputs = self.model( 2025-12-04T09:50:03.6128241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6128641Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6129007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6129433Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6129845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6130233Z return func(*args, **kwargs) 2025-12-04T09:50:03.6130604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6131003Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6131148Z 2025-12-04T09:50:03.6131256Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6131620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6131940Z res = mod(**inputs) 2025-12-04T09:50:03.6132317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6132699Z outputs = self.model( 2025-12-04T09:50:03.6133056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6133450Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6133813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6134180Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6134560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6134936Z return func(*args, **kwargs) 2025-12-04T09:50:03.6135309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:50:03.6135705Z hidden_states = residual + hidden_states 2025-12-04T09:50:03.6135851Z 2025-12-04T09:50:03.6135958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6136322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6136644Z res = mod(**inputs) 2025-12-04T09:50:03.6136996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6137383Z outputs = self.model( 2025-12-04T09:50:03.6137746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6138134Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6138488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6138879Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6139277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6139647Z return func(*args, **kwargs) 2025-12-04T09:50:03.6140021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6140435Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6140855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6141247Z return func(*args, **kwargs) 2025-12-04T09:50:03.6141626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6142061Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6142229Z 2025-12-04T09:50:03.6142343Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6142700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6143030Z res = mod(**inputs) 2025-12-04T09:50:03.6143392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6143773Z outputs = self.model( 2025-12-04T09:50:03.6144159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6144554Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6144909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6145274Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6145663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6146044Z return func(*args, **kwargs) 2025-12-04T09:50:03.6146443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6146850Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6147394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6147777Z return func(*args, **kwargs) 2025-12-04T09:50:03.6148151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6148551Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6148689Z 2025-12-04T09:50:03.6148802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6149156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6149481Z res = mod(**inputs) 2025-12-04T09:50:03.6149842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6150233Z outputs = self.model( 2025-12-04T09:50:03.6150580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6150960Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6151310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6151669Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6152037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6152408Z return func(*args, **kwargs) 2025-12-04T09:50:03.6152771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6153169Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6153568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6153935Z return func(*args, **kwargs) 2025-12-04T09:50:03.6154298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6154708Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6154927Z 2025-12-04T09:50:03.6155057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6155423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6155745Z res = mod(**inputs) 2025-12-04T09:50:03.6156108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6156518Z outputs = self.model( 2025-12-04T09:50:03.6156912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6157331Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6157719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6158123Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6158575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6158983Z return func(*args, **kwargs) 2025-12-04T09:50:03.6159392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6159844Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6160282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6160757Z return func(*args, **kwargs) 2025-12-04T09:50:03.6161202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6161694Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6161905Z 2025-12-04T09:50:03.6162012Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6162378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6162710Z res = mod(**inputs) 2025-12-04T09:50:03.6163073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6163491Z outputs = self.model( 2025-12-04T09:50:03.6163889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6164308Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6164680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6165078Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6165492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6165900Z return func(*args, **kwargs) 2025-12-04T09:50:03.6166302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6166749Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6167183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6167575Z return func(*args, **kwargs) 2025-12-04T09:50:03.6167982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6168425Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6168579Z 2025-12-04T09:50:03.6168699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6169079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6169427Z res = mod(**inputs) 2025-12-04T09:50:03.6169812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6170247Z outputs = self.model( 2025-12-04T09:50:03.6170636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6171054Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6171429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6171807Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6172241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6172621Z return func(*args, **kwargs) 2025-12-04T09:50:03.6173003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6173414Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6173816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6174188Z return func(*args, **kwargs) 2025-12-04T09:50:03.6174545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6174949Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6175106Z 2025-12-04T09:50:03.6175210Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6175582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6175892Z res = mod(**inputs) 2025-12-04T09:50:03.6176247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6176628Z outputs = self.model( 2025-12-04T09:50:03.6176986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6177382Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6177738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6178104Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6178484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6178869Z return func(*args, **kwargs) 2025-12-04T09:50:03.6179246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6179660Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6180061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6180441Z return func(*args, **kwargs) 2025-12-04T09:50:03.6180817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6181256Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6181445Z 2025-12-04T09:50:03.6181551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6181914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6182238Z res = mod(**inputs) 2025-12-04T09:50:03.6182591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6182981Z outputs = self.model( 2025-12-04T09:50:03.6183348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6183735Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6184099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6184510Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6184896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6185265Z return func(*args, **kwargs) 2025-12-04T09:50:03.6185639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6186069Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6186506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6186877Z return func(*args, **kwargs) 2025-12-04T09:50:03.6187270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6187672Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6187815Z 2025-12-04T09:50:03.6187924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6188294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6188624Z res = mod(**inputs) 2025-12-04T09:50:03.6188985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6189359Z outputs = self.model( 2025-12-04T09:50:03.6189724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6190137Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6190489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6190859Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6191245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6191624Z return func(*args, **kwargs) 2025-12-04T09:50:03.6191987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6192421Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6192597Z 2025-12-04T09:50:03.6192718Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6193102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6193440Z res = mod(**inputs) 2025-12-04T09:50:03.6193819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6194222Z outputs = self.model( 2025-12-04T09:50:03.6194599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6195013Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6195390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6195775Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6196171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6196572Z return func(*args, **kwargs) 2025-12-04T09:50:03.6196966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6197440Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6197858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6198231Z return self.act(input) 2025-12-04T09:50:03.6198349Z 2025-12-04T09:50:03.6198490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6198884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6199233Z res = mod(**inputs) 2025-12-04T09:50:03.6199613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6200030Z outputs = self.model( 2025-12-04T09:50:03.6200410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6200920Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6201304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6201690Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6202134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6202534Z return func(*args, **kwargs) 2025-12-04T09:50:03.6202907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6203297Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6203444Z 2025-12-04T09:50:03.6204138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6204502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6204846Z res = mod(**inputs) 2025-12-04T09:50:03.6205213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6205598Z outputs = self.model( 2025-12-04T09:50:03.6205961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6206341Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6206699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6207072Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6207458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6207832Z return func(*args, **kwargs) 2025-12-04T09:50:03.6208208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6208624Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6209023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6209412Z return func(*args, **kwargs) 2025-12-04T09:50:03.6209778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6210198Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6210362Z 2025-12-04T09:50:03.6210466Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6210821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6211141Z res = mod(**inputs) 2025-12-04T09:50:03.6211485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6211859Z outputs = self.model( 2025-12-04T09:50:03.6212215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6212591Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6212932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6213316Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6213710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6214077Z return func(*args, **kwargs) 2025-12-04T09:50:03.6214444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6214866Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6215293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6215687Z return func(*args, **kwargs) 2025-12-04T09:50:03.6216077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6216509Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6216654Z 2025-12-04T09:50:03.6216809Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6217170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6217501Z res = mod(**inputs) 2025-12-04T09:50:03.6217861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6218235Z outputs = self.model( 2025-12-04T09:50:03.6218598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6219005Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6219361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6219723Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6220111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6220495Z return func(*args, **kwargs) 2025-12-04T09:50:03.6220861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6221290Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6221686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6222059Z return func(*args, **kwargs) 2025-12-04T09:50:03.6222421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6222853Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6223029Z 2025-12-04T09:50:03.6223135Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6223501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6223823Z res = mod(**inputs) 2025-12-04T09:50:03.6224186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6224572Z outputs = self.model( 2025-12-04T09:50:03.6224926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6225317Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6225672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6226042Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6226421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6226797Z return func(*args, **kwargs) 2025-12-04T09:50:03.6227175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6227635Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6228055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6228439Z return func(*args, **kwargs) 2025-12-04T09:50:03.6228819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6229270Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6229472Z 2025-12-04T09:50:03.6229579Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6229945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6230261Z res = mod(**inputs) 2025-12-04T09:50:03.6230623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6231006Z outputs = self.model( 2025-12-04T09:50:03.6231361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6231732Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6232079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6232438Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6232814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6233240Z return func(*args, **kwargs) 2025-12-04T09:50:03.6233614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6234030Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6234441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6234818Z return func(*args, **kwargs) 2025-12-04T09:50:03.6235194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6235599Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6235746Z 2025-12-04T09:50:03.6235851Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6236217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6236545Z res = mod(**inputs) 2025-12-04T09:50:03.6236906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6237284Z outputs = self.model( 2025-12-04T09:50:03.6237651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6238050Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6238424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6238821Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6239232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6239637Z return func(*args, **kwargs) 2025-12-04T09:50:03.6240027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6240536Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6240982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6241391Z return func(*args, **kwargs) 2025-12-04T09:50:03.6241784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6242261Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6242420Z 2025-12-04T09:50:03.6242536Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6242896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6243224Z res = mod(**inputs) 2025-12-04T09:50:03.6243590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6243977Z outputs = self.model( 2025-12-04T09:50:03.6244337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6244727Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6245101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6245466Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6245850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6246224Z return func(*args, **kwargs) 2025-12-04T09:50:03.6246603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6247010Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6247628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6248091Z return func(*args, **kwargs) 2025-12-04T09:50:03.6248480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6248969Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6249168Z 2025-12-04T09:50:03.6249281Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6249669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6250007Z res = mod(**inputs) 2025-12-04T09:50:03.6250399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6250815Z outputs = self.model( 2025-12-04T09:50:03.6251213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6251619Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6251975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6252341Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6252725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6253105Z return func(*args, **kwargs) 2025-12-04T09:50:03.6253480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6253896Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6254293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6254670Z return func(*args, **kwargs) 2025-12-04T09:50:03.6255048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6255450Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6255589Z 2025-12-04T09:50:03.6255694Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6256059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6256418Z res = mod(**inputs) 2025-12-04T09:50:03.6256837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6257265Z outputs = self.model( 2025-12-04T09:50:03.6257654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6258064Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6258417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6258791Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6259179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6259555Z return func(*args, **kwargs) 2025-12-04T09:50:03.6259958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6260402Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6260575Z 2025-12-04T09:50:03.6260690Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6261043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6261368Z res = mod(**inputs) 2025-12-04T09:50:03.6261726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6262123Z outputs = self.model( 2025-12-04T09:50:03.6262476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6262864Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6263216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6263577Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6263965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6264347Z return func(*args, **kwargs) 2025-12-04T09:50:03.6264726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6265152Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6265547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6265901Z return self.act(input) 2025-12-04T09:50:03.6266015Z 2025-12-04T09:50:03.6266120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6266509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6266888Z res = mod(**inputs) 2025-12-04T09:50:03.6267266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6267642Z outputs = self.model( 2025-12-04T09:50:03.6268003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6268393Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6268745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6269116Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6269502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6269881Z return func(*args, **kwargs) 2025-12-04T09:50:03.6270261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6270686Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6270847Z 2025-12-04T09:50:03.6270991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6271363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6271702Z res = mod(**inputs) 2025-12-04T09:50:03.6272083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6272491Z outputs = self.model( 2025-12-04T09:50:03.6272861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6273256Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6273613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6274003Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6274430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6274840Z return func(*args, **kwargs) 2025-12-04T09:50:03.6275241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:50:03.6275659Z hidden_states = residual + hidden_states 2025-12-04T09:50:03.6275817Z 2025-12-04T09:50:03.6275928Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6276314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6276675Z res = mod(**inputs) 2025-12-04T09:50:03.6277048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6277451Z outputs = self.model( 2025-12-04T09:50:03.6277834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6278239Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6278613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6279002Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6279408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6279811Z return func(*args, **kwargs) 2025-12-04T09:50:03.6280204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6280740Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6281197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6281614Z return func(*args, **kwargs) 2025-12-04T09:50:03.6282048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6282519Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6282698Z 2025-12-04T09:50:03.6282813Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6283207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6283560Z res = mod(**inputs) 2025-12-04T09:50:03.6283950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6284360Z outputs = self.model( 2025-12-04T09:50:03.6284751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6285187Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6285565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6285994Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6286406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6286814Z return func(*args, **kwargs) 2025-12-04T09:50:03.6287201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6287632Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6288027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6288403Z return func(*args, **kwargs) 2025-12-04T09:50:03.6288771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6289192Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6289334Z 2025-12-04T09:50:03.6289450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6289808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6290143Z res = mod(**inputs) 2025-12-04T09:50:03.6290497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6290880Z outputs = self.model( 2025-12-04T09:50:03.6291233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6291638Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6291993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6292349Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6292728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6293101Z return func(*args, **kwargs) 2025-12-04T09:50:03.6293472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6293872Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6294269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6294642Z return func(*args, **kwargs) 2025-12-04T09:50:03.6295013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6295428Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6295602Z 2025-12-04T09:50:03.6295708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6296084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6296399Z res = mod(**inputs) 2025-12-04T09:50:03.6296755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6297134Z outputs = self.model( 2025-12-04T09:50:03.6297497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6297883Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6298241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6298614Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6298999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6299381Z return func(*args, **kwargs) 2025-12-04T09:50:03.6299758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6300206Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6300610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6300980Z return func(*args, **kwargs) 2025-12-04T09:50:03.6301343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6301798Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6301991Z 2025-12-04T09:50:03.6302096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6302476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6302809Z res = mod(**inputs) 2025-12-04T09:50:03.6303197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6303584Z outputs = self.model( 2025-12-04T09:50:03.6303944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6304331Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6304675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6305037Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6305417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6305797Z return func(*args, **kwargs) 2025-12-04T09:50:03.6306166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6306569Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6306967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6307327Z return func(*args, **kwargs) 2025-12-04T09:50:03.6307695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6308091Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6308233Z 2025-12-04T09:50:03.6308344Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6308693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6309013Z res = mod(**inputs) 2025-12-04T09:50:03.6309371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6309748Z outputs = self.model( 2025-12-04T09:50:03.6310124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6310510Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6310870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6311236Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6311623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6312012Z return func(*args, **kwargs) 2025-12-04T09:50:03.6312380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6312799Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6313209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6313599Z return func(*args, **kwargs) 2025-12-04T09:50:03.6314006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6314418Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6314570Z 2025-12-04T09:50:03.6314683Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6315045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6315354Z res = mod(**inputs) 2025-12-04T09:50:03.6315711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6316097Z outputs = self.model( 2025-12-04T09:50:03.6316452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6316843Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6317215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6317587Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6317966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6318339Z return func(*args, **kwargs) 2025-12-04T09:50:03.6318714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6318817Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6319090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6319161Z return func(*args, **kwargs) 2025-12-04T09:50:03.6319410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6319548Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6319554Z 2025-12-04T09:50:03.6319659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6319857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6319928Z res = mod(**inputs) 2025-12-04T09:50:03.6320175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6320251Z outputs = self.model( 2025-12-04T09:50:03.6320574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6320666Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6320916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6321003Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6321264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6321352Z return func(*args, **kwargs) 2025-12-04T09:50:03.6321621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6321736Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6322000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6322071Z return func(*args, **kwargs) 2025-12-04T09:50:03.6322335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6322418Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6322422Z 2025-12-04T09:50:03.6322532Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6322736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6322855Z res = mod(**inputs) 2025-12-04T09:50:03.6323113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6323181Z outputs = self.model( 2025-12-04T09:50:03.6323425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6323509Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6323733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6323820Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6324062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6324132Z return func(*args, **kwargs) 2025-12-04T09:50:03.6324403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6324530Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6324534Z 2025-12-04T09:50:03.6324644Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6324844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6324908Z res = mod(**inputs) 2025-12-04T09:50:03.6325165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6325254Z outputs = self.model( 2025-12-04T09:50:03.6325507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6325590Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6325822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6325912Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6326160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6326229Z return func(*args, **kwargs) 2025-12-04T09:50:03.6326493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6326616Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6326839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6326918Z return self.act(input) 2025-12-04T09:50:03.6326922Z 2025-12-04T09:50:03.6327025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6327233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6327301Z res = mod(**inputs) 2025-12-04T09:50:03.6327569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6327646Z outputs = self.model( 2025-12-04T09:50:03.6327898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6327981Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6328210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6328292Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6328547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6328617Z return func(*args, **kwargs) 2025-12-04T09:50:03.6328868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6328994Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6328999Z 2025-12-04T09:50:03.6329106Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6329311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6329375Z res = mod(**inputs) 2025-12-04T09:50:03.6329622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6329699Z outputs = self.model( 2025-12-04T09:50:03.6329949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6330023Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6330253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6330348Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6330606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6330676Z return func(*args, **kwargs) 2025-12-04T09:50:03.6330925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6331035Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6331276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6331390Z return func(*args, **kwargs) 2025-12-04T09:50:03.6331644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6331763Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6331766Z 2025-12-04T09:50:03.6331880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6332084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6332149Z res = mod(**inputs) 2025-12-04T09:50:03.6332408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6332479Z outputs = self.model( 2025-12-04T09:50:03.6332739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6332815Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6333043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6333132Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6333379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6333456Z return func(*args, **kwargs) 2025-12-04T09:50:03.6333711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6333813Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6334067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6334138Z return func(*args, **kwargs) 2025-12-04T09:50:03.6334393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6334482Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6334485Z 2025-12-04T09:50:03.6334589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6334802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6334867Z res = mod(**inputs) 2025-12-04T09:50:03.6335153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6335229Z outputs = self.model( 2025-12-04T09:50:03.6335470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6335541Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6335766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6335844Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6336093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6336162Z return func(*args, **kwargs) 2025-12-04T09:50:03.6336429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6336538Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6336780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6336857Z return func(*args, **kwargs) 2025-12-04T09:50:03.6337104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6337217Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6337220Z 2025-12-04T09:50:03.6337350Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6337551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6337616Z res = mod(**inputs) 2025-12-04T09:50:03.6337871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6337942Z outputs = self.model( 2025-12-04T09:50:03.6338202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6338276Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6338503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6338591Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6338838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6338918Z return func(*args, **kwargs) 2025-12-04T09:50:03.6339170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6339267Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6339523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6339595Z return func(*args, **kwargs) 2025-12-04T09:50:03.6339847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6339992Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6339996Z 2025-12-04T09:50:03.6340099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6340314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6340380Z res = mod(**inputs) 2025-12-04T09:50:03.6340625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6340700Z outputs = self.model( 2025-12-04T09:50:03.6340947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6341018Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6341280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6341365Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6341610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6341680Z return func(*args, **kwargs) 2025-12-04T09:50:03.6341923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6342029Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6342267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6342341Z return func(*args, **kwargs) 2025-12-04T09:50:03.6343330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6343431Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6343436Z 2025-12-04T09:50:03.6343543Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6343736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6343801Z res = mod(**inputs) 2025-12-04T09:50:03.6344051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6344149Z outputs = self.model( 2025-12-04T09:50:03.6344401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6344474Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6344703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6344792Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6345043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6345121Z return func(*args, **kwargs) 2025-12-04T09:50:03.6345374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6345474Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6345733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6345805Z return func(*args, **kwargs) 2025-12-04T09:50:03.6346057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6346165Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6346168Z 2025-12-04T09:50:03.6346277Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6346499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6346568Z res = mod(**inputs) 2025-12-04T09:50:03.6346835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6346914Z outputs = self.model( 2025-12-04T09:50:03.6347355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6347445Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6347693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6347776Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6348042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6348173Z return func(*args, **kwargs) 2025-12-04T09:50:03.6348451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6348558Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6348794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6348868Z return func(*args, **kwargs) 2025-12-04T09:50:03.6349108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6349235Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6349238Z 2025-12-04T09:50:03.6349346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6349569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6349637Z res = mod(**inputs) 2025-12-04T09:50:03.6349902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6349973Z outputs = self.model( 2025-12-04T09:50:03.6350232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6350308Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6350538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6350660Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6350899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6350973Z return func(*args, **kwargs) 2025-12-04T09:50:03.6351218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6351318Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6351561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6351628Z return func(*args, **kwargs) 2025-12-04T09:50:03.6351872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6351961Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6351965Z 2025-12-04T09:50:03.6352070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6352274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6352339Z res = mod(**inputs) 2025-12-04T09:50:03.6352590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6352667Z outputs = self.model( 2025-12-04T09:50:03.6352920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6352992Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6353230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6353307Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6353553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6353623Z return func(*args, **kwargs) 2025-12-04T09:50:03.6353864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6353990Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6353994Z 2025-12-04T09:50:03.6354095Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6354331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6354397Z res = mod(**inputs) 2025-12-04T09:50:03.6354638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6354713Z outputs = self.model( 2025-12-04T09:50:03.6354954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6355029Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6355255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6355334Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6355599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6355671Z return func(*args, **kwargs) 2025-12-04T09:50:03.6355925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6356053Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6356283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6356365Z return self.act(input) 2025-12-04T09:50:03.6356369Z 2025-12-04T09:50:03.6356478Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6356708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6356786Z res = mod(**inputs) 2025-12-04T09:50:03.6357051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6357125Z outputs = self.model( 2025-12-04T09:50:03.6357402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6357482Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6357725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6357811Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6358073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6358155Z return func(*args, **kwargs) 2025-12-04T09:50:03.6358422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6358510Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6358523Z 2025-12-04T09:50:03.6358631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6358842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6358922Z res = mod(**inputs) 2025-12-04T09:50:03.6359200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6359272Z outputs = self.model( 2025-12-04T09:50:03.6359546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6359624Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6359868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6359955Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6360214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6360295Z return func(*args, **kwargs) 2025-12-04T09:50:03.6360620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:50:03.6360758Z hidden_states = residual + hidden_states 2025-12-04T09:50:03.6360763Z 2025-12-04T09:50:03.6360884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6361103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6361180Z res = mod(**inputs) 2025-12-04T09:50:03.6361454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6361529Z outputs = self.model( 2025-12-04T09:50:03.6361815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6361906Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6362170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6362268Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6362536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6362615Z return func(*args, **kwargs) 2025-12-04T09:50:03.6362866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6362968Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6363217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6363306Z return func(*args, **kwargs) 2025-12-04T09:50:03.6363564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6363682Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6363686Z 2025-12-04T09:50:03.6363791Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6363997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6364062Z res = mod(**inputs) 2025-12-04T09:50:03.6364309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6364383Z outputs = self.model( 2025-12-04T09:50:03.6364637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6364718Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6364939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6365018Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6365268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6365340Z return func(*args, **kwargs) 2025-12-04T09:50:03.6365587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6365692Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6365935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6366009Z return func(*args, **kwargs) 2025-12-04T09:50:03.6366258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6366339Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6366342Z 2025-12-04T09:50:03.6366453Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6366650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6366738Z res = mod(**inputs) 2025-12-04T09:50:03.6367013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6367087Z outputs = self.model( 2025-12-04T09:50:03.6367356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6367434Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6367678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6367764Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6368006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6368082Z return func(*args, **kwargs) 2025-12-04T09:50:03.6368370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6368482Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6368757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6368826Z return func(*args, **kwargs) 2025-12-04T09:50:03.6369084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6369195Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6369215Z 2025-12-04T09:50:03.6369320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6369525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6369591Z res = mod(**inputs) 2025-12-04T09:50:03.6369842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6369918Z outputs = self.model( 2025-12-04T09:50:03.6370166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6370249Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6370471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6370550Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6370806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6370877Z return func(*args, **kwargs) 2025-12-04T09:50:03.6371126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6371233Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6371476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6371555Z return func(*args, **kwargs) 2025-12-04T09:50:03.6371808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6371946Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6371950Z 2025-12-04T09:50:03.6372063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6372264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6372339Z res = mod(**inputs) 2025-12-04T09:50:03.6372586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6372654Z outputs = self.model( 2025-12-04T09:50:03.6372910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6373006Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6373256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6373344Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6373588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6373665Z return func(*args, **kwargs) 2025-12-04T09:50:03.6373920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6374021Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6374273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6374357Z return func(*args, **kwargs) 2025-12-04T09:50:03.6374617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6374707Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6374711Z 2025-12-04T09:50:03.6374818Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6375030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6375096Z res = mod(**inputs) 2025-12-04T09:50:03.6375344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6375438Z outputs = self.model( 2025-12-04T09:50:03.6375687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6375768Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6375995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6376077Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6376332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6376401Z return func(*args, **kwargs) 2025-12-04T09:50:03.6376670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6376774Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6377019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6377093Z return func(*args, **kwargs) 2025-12-04T09:50:03.6377363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6377462Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6377468Z 2025-12-04T09:50:03.6377581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6377776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6377848Z res = mod(**inputs) 2025-12-04T09:50:03.6378099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6378168Z outputs = self.model( 2025-12-04T09:50:03.6378432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6378508Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6378732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6378819Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6379069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6379184Z return func(*args, **kwargs) 2025-12-04T09:50:03.6379437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6379536Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6379790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6379859Z return func(*args, **kwargs) 2025-12-04T09:50:03.6380116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6380246Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6380250Z 2025-12-04T09:50:03.6380353Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6380573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6380641Z res = mod(**inputs) 2025-12-04T09:50:03.6380909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6380987Z outputs = self.model( 2025-12-04T09:50:03.6381236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6381317Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6381543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6381642Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6381891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6381959Z return func(*args, **kwargs) 2025-12-04T09:50:03.6382217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6382319Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6382564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6382640Z return func(*args, **kwargs) 2025-12-04T09:50:03.6382889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6382970Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6382975Z 2025-12-04T09:50:03.6383084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6383283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6383353Z res = mod(**inputs) 2025-12-04T09:50:03.6383605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6383677Z outputs = self.model( 2025-12-04T09:50:03.6383938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6384012Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6384240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6384326Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6384571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6384648Z return func(*args, **kwargs) 2025-12-04T09:50:03.6384898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6385020Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6385023Z 2025-12-04T09:50:03.6385157Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6385368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6385443Z res = mod(**inputs) 2025-12-04T09:50:03.6385693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6385762Z outputs = self.model( 2025-12-04T09:50:03.6386015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6386092Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6386319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6386405Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6386666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6386745Z return func(*args, **kwargs) 2025-12-04T09:50:03.6386999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6387120Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6387349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6387419Z return self.act(input) 2025-12-04T09:50:03.6387422Z 2025-12-04T09:50:03.6387550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6387746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6387813Z res = mod(**inputs) 2025-12-04T09:50:03.6388073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6388141Z outputs = self.model( 2025-12-04T09:50:03.6388400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6388487Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6388720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6388811Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6389071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6389147Z return func(*args, **kwargs) 2025-12-04T09:50:03.6389416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6389502Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6389506Z 2025-12-04T09:50:03.6389615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6389836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6389903Z res = mod(**inputs) 2025-12-04T09:50:03.6390172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6390254Z outputs = self.model( 2025-12-04T09:50:03.6390498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6390580Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6390802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6390890Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6391132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6391204Z return func(*args, **kwargs) 2025-12-04T09:50:03.6391503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6391605Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6391852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6391929Z return func(*args, **kwargs) 2025-12-04T09:50:03.6392180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6392301Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6392305Z 2025-12-04T09:50:03.6392408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6392608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6392697Z res = mod(**inputs) 2025-12-04T09:50:03.6392964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6393039Z outputs = self.model( 2025-12-04T09:50:03.6393321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6393402Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6393651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6393734Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6394023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6394105Z return func(*args, **kwargs) 2025-12-04T09:50:03.6394371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6394483Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6394749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6394822Z return func(*args, **kwargs) 2025-12-04T09:50:03.6395091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6395187Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6395191Z 2025-12-04T09:50:03.6395294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6395500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6395564Z res = mod(**inputs) 2025-12-04T09:50:03.6395820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6395890Z outputs = self.model( 2025-12-04T09:50:03.6396140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6396225Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6396448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6396526Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6396781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6396851Z return func(*args, **kwargs) 2025-12-04T09:50:03.6397135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6397240Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6397509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6397608Z return func(*args, **kwargs) 2025-12-04T09:50:03.6397900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6398026Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6398030Z 2025-12-04T09:50:03.6398142Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6398354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6398430Z res = mod(**inputs) 2025-12-04T09:50:03.6398698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6398770Z outputs = self.model( 2025-12-04T09:50:03.6399051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6399145Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6399399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6399484Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6399757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6399838Z return func(*args, **kwargs) 2025-12-04T09:50:03.6400146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6400281Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6400630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6400715Z return func(*args, **kwargs) 2025-12-04T09:50:03.6401004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6401159Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6401164Z 2025-12-04T09:50:03.6401281Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6401509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6401581Z res = mod(**inputs) 2025-12-04T09:50:03.6401864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6401942Z outputs = self.model( 2025-12-04T09:50:03.6402209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6402294Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6402519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6402609Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6402858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6402929Z return func(*args, **kwargs) 2025-12-04T09:50:03.6403187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6403288Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6403534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6403615Z return func(*args, **kwargs) 2025-12-04T09:50:03.6403867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6403965Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6403968Z 2025-12-04T09:50:03.6404076Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6404313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6404390Z res = mod(**inputs) 2025-12-04T09:50:03.6404641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6404711Z outputs = self.model( 2025-12-04T09:50:03.6404964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6405041Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6405273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6405352Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6405612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6405691Z return func(*args, **kwargs) 2025-12-04T09:50:03.6405940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6406047Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6406288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6406357Z return func(*args, **kwargs) 2025-12-04T09:50:03.6406611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6406727Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6406730Z 2025-12-04T09:50:03.6406833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6407037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6407104Z res = mod(**inputs) 2025-12-04T09:50:03.6407363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6407430Z outputs = self.model( 2025-12-04T09:50:03.6407689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6407776Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6408015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6408100Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6408364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6408436Z return func(*args, **kwargs) 2025-12-04T09:50:03.6408709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6408815Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6409071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6409152Z return func(*args, **kwargs) 2025-12-04T09:50:03.6409415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6409571Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6409574Z 2025-12-04T09:50:03.6409679Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6409876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6409950Z res = mod(**inputs) 2025-12-04T09:50:03.6410198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6410268Z outputs = self.model( 2025-12-04T09:50:03.6410558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6410634Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6410866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6410948Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6411192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6411270Z return func(*args, **kwargs) 2025-12-04T09:50:03.6411521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6411626Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6411888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6411962Z return func(*args, **kwargs) 2025-12-04T09:50:03.6412234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6412321Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6412325Z 2025-12-04T09:50:03.6412437Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6412656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6412742Z res = mod(**inputs) 2025-12-04T09:50:03.6413017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6413089Z outputs = self.model( 2025-12-04T09:50:03.6413358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6413445Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6413689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6413773Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6414041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6414112Z return func(*args, **kwargs) 2025-12-04T09:50:03.6414373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6414495Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6414499Z 2025-12-04T09:50:03.6414602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6414813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6414879Z res = mod(**inputs) 2025-12-04T09:50:03.6415143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6415214Z outputs = self.model( 2025-12-04T09:50:03.6415467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6415548Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6415778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6415858Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6416117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6416185Z return func(*args, **kwargs) 2025-12-04T09:50:03.6416446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6416567Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6416829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6416913Z return self.act(input) 2025-12-04T09:50:03.6416916Z 2025-12-04T09:50:03.6417020Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6417223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6417289Z res = mod(**inputs) 2025-12-04T09:50:03.6417538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6417616Z outputs = self.model( 2025-12-04T09:50:03.6417863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6417936Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6418185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6418271Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6418528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6418598Z return func(*args, **kwargs) 2025-12-04T09:50:03.6418850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6418941Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6418962Z 2025-12-04T09:50:03.6419070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6419268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6419344Z res = mod(**inputs) 2025-12-04T09:50:03.6419593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6419670Z outputs = self.model( 2025-12-04T09:50:03.6419921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6419998Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6420229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6420311Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6420561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6420633Z return func(*args, **kwargs) 2025-12-04T09:50:03.6420880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:50:03.6420968Z hidden_states = residual + hidden_states 2025-12-04T09:50:03.6420972Z 2025-12-04T09:50:03.6421074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6421274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6421345Z res = mod(**inputs) 2025-12-04T09:50:03.6421592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6421666Z outputs = self.model( 2025-12-04T09:50:03.6421915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6421991Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6422223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6422313Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6422550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6422644Z return func(*args, **kwargs) 2025-12-04T09:50:03.6422898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6423004Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6423243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6423310Z return func(*args, **kwargs) 2025-12-04T09:50:03.6423563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6423677Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6423681Z 2025-12-04T09:50:03.6423790Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6424001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6424068Z res = mod(**inputs) 2025-12-04T09:50:03.6424319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6424386Z outputs = self.model( 2025-12-04T09:50:03.6424629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6424709Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6424929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6425034Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6425276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6425345Z return func(*args, **kwargs) 2025-12-04T09:50:03.6425597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6425699Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6425945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6426011Z return func(*args, **kwargs) 2025-12-04T09:50:03.6426252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6426339Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6426344Z 2025-12-04T09:50:03.6426442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6426636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6426705Z res = mod(**inputs) 2025-12-04T09:50:03.6426953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6427029Z outputs = self.model( 2025-12-04T09:50:03.6427273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6427346Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6427573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6427650Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6427886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6427963Z return func(*args, **kwargs) 2025-12-04T09:50:03.6428209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6428313Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6428558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6428662Z return func(*args, **kwargs) 2025-12-04T09:50:03.6428922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6429041Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6429045Z 2025-12-04T09:50:03.6429158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6429367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6429437Z res = mod(**inputs) 2025-12-04T09:50:03.6429704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6429777Z outputs = self.model( 2025-12-04T09:50:03.6430070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6430162Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6430401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6430491Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6430755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6430824Z return func(*args, **kwargs) 2025-12-04T09:50:03.6431081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6431197Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6431449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6431523Z return func(*args, **kwargs) 2025-12-04T09:50:03.6431788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6431945Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6431949Z 2025-12-04T09:50:03.6432068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6432267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6432339Z res = mod(**inputs) 2025-12-04T09:50:03.6432591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6432670Z outputs = self.model( 2025-12-04T09:50:03.6432918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6432993Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6433224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6433311Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6433571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6433651Z return func(*args, **kwargs) 2025-12-04T09:50:03.6433918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6434031Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6434289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6434365Z return func(*args, **kwargs) 2025-12-04T09:50:03.6434636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6434730Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6434734Z 2025-12-04T09:50:03.6434869Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6435101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6435171Z res = mod(**inputs) 2025-12-04T09:50:03.6435457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6435532Z outputs = self.model( 2025-12-04T09:50:03.6435802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6435886Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6436112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6436197Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6436469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6436546Z return func(*args, **kwargs) 2025-12-04T09:50:03.6436818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6436923Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6437189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6437262Z return func(*args, **kwargs) 2025-12-04T09:50:03.6437526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6437666Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6437670Z 2025-12-04T09:50:03.6437778Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6437990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6438068Z res = mod(**inputs) 2025-12-04T09:50:03.6438332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6438412Z outputs = self.model( 2025-12-04T09:50:03.6438673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6438752Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6438999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6439087Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6439352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6439435Z return func(*args, **kwargs) 2025-12-04T09:50:03.6439704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6439824Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6440099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6440173Z return func(*args, **kwargs) 2025-12-04T09:50:03.6440451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6440672Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6440681Z 2025-12-04T09:50:03.6440803Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6441023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6441095Z res = mod(**inputs) 2025-12-04T09:50:03.6441379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6441478Z outputs = self.model( 2025-12-04T09:50:03.6441768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6441861Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6442114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6442209Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6442465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6442540Z return func(*args, **kwargs) 2025-12-04T09:50:03.6442817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6442923Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6443203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6443282Z return func(*args, **kwargs) 2025-12-04T09:50:03.6443547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6443645Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6443648Z 2025-12-04T09:50:03.6443760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6443970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6444066Z res = mod(**inputs) 2025-12-04T09:50:03.6444335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6444413Z outputs = self.model( 2025-12-04T09:50:03.6444679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6444760Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6445010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6445094Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6445356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6445437Z return func(*args, **kwargs) 2025-12-04T09:50:03.6445707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6445845Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6445849Z 2025-12-04T09:50:03.6445959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6446171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6446248Z res = mod(**inputs) 2025-12-04T09:50:03.6446515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6446594Z outputs = self.model( 2025-12-04T09:50:03.6446862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6446943Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6447355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6447448Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6447712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6447795Z return func(*args, **kwargs) 2025-12-04T09:50:03.6448067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6448266Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6448499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6448576Z return self.act(input) 2025-12-04T09:50:03.6448581Z 2025-12-04T09:50:03.6448699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6448906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6448984Z res = mod(**inputs) 2025-12-04T09:50:03.6449249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6449322Z outputs = self.model( 2025-12-04T09:50:03.6449592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6449703Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6449946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6450040Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6450299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6450380Z return func(*args, **kwargs) 2025-12-04T09:50:03.6450640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6450753Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6450757Z 2025-12-04T09:50:03.6450874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6451086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6451154Z res = mod(**inputs) 2025-12-04T09:50:03.6451425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6451499Z outputs = self.model( 2025-12-04T09:50:03.6451769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6451846Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6452081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6452172Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6452432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6452512Z return func(*args, **kwargs) 2025-12-04T09:50:03.6452774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6452879Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6453144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6453217Z return func(*args, **kwargs) 2025-12-04T09:50:03.6453476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6453601Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6453605Z 2025-12-04T09:50:03.6453714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6453942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6454007Z res = mod(**inputs) 2025-12-04T09:50:03.6454256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6454334Z outputs = self.model( 2025-12-04T09:50:03.6454601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6454694Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6454929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6455010Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6455267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6455336Z return func(*args, **kwargs) 2025-12-04T09:50:03.6455592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6455699Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6455963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6456042Z return func(*args, **kwargs) 2025-12-04T09:50:03.6456293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6456380Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6456384Z 2025-12-04T09:50:03.6456500Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6456706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6456775Z res = mod(**inputs) 2025-12-04T09:50:03.6457069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6457137Z outputs = self.model( 2025-12-04T09:50:03.6457394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6457468Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6457694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6457781Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6458025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6458103Z return func(*args, **kwargs) 2025-12-04T09:50:03.6458351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6458449Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6458698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6458766Z return func(*args, **kwargs) 2025-12-04T09:50:03.6459018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6459135Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6459140Z 2025-12-04T09:50:03.6459244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6459449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6459513Z res = mod(**inputs) 2025-12-04T09:50:03.6459763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6459840Z outputs = self.model( 2025-12-04T09:50:03.6460089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6460161Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6460392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6460471Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6460762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6460833Z return func(*args, **kwargs) 2025-12-04T09:50:03.6461081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6461189Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6461433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6461509Z return func(*args, **kwargs) 2025-12-04T09:50:03.6461756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6461892Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6461895Z 2025-12-04T09:50:03.6462033Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6462235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6462300Z res = mod(**inputs) 2025-12-04T09:50:03.6462555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6462624Z outputs = self.model( 2025-12-04T09:50:03.6462880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6462955Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6463196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6463282Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6463529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6463612Z return func(*args, **kwargs) 2025-12-04T09:50:03.6463867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6463969Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6464225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6464296Z return func(*args, **kwargs) 2025-12-04T09:50:03.6464548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6464652Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6464655Z 2025-12-04T09:50:03.6464760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6464966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6465036Z res = mod(**inputs) 2025-12-04T09:50:03.6465289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6465371Z outputs = self.model( 2025-12-04T09:50:03.6465623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6465700Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6465929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6466011Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6466266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6466336Z return func(*args, **kwargs) 2025-12-04T09:50:03.6466587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6466697Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6466976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6467053Z return func(*args, **kwargs) 2025-12-04T09:50:03.6467303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6467400Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6467403Z 2025-12-04T09:50:03.6467515Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6467716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6467781Z res = mod(**inputs) 2025-12-04T09:50:03.6468040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6468130Z outputs = self.model( 2025-12-04T09:50:03.6468390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6468464Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6468688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6468774Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6469021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6469117Z return func(*args, **kwargs) 2025-12-04T09:50:03.6469370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6469469Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6469725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6469800Z return func(*args, **kwargs) 2025-12-04T09:50:03.6470070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6470218Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6470222Z 2025-12-04T09:50:03.6470331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6470548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6470618Z res = mod(**inputs) 2025-12-04T09:50:03.6470885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6470967Z outputs = self.model( 2025-12-04T09:50:03.6471232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6471311Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6471565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6471648Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6471912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6471981Z return func(*args, **kwargs) 2025-12-04T09:50:03.6472230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6472337Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6472581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6472657Z return func(*args, **kwargs) 2025-12-04T09:50:03.6472906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6473012Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6473031Z 2025-12-04T09:50:03.6473144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6473342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6473406Z res = mod(**inputs) 2025-12-04T09:50:03.6473676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6473748Z outputs = self.model( 2025-12-04T09:50:03.6474022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6474101Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6474341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6474445Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6474694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6474770Z return func(*args, **kwargs) 2025-12-04T09:50:03.6475019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6475140Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6475144Z 2025-12-04T09:50:03.6475253Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6475472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6475536Z res = mod(**inputs) 2025-12-04T09:50:03.6475793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6475860Z outputs = self.model( 2025-12-04T09:50:03.6476119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6476196Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6476421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6476508Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6476753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6476821Z return func(*args, **kwargs) 2025-12-04T09:50:03.6477079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6477199Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6477429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6477500Z return self.act(input) 2025-12-04T09:50:03.6477505Z 2025-12-04T09:50:03.6477610Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6477817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6477886Z res = mod(**inputs) 2025-12-04T09:50:03.6478157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6478231Z outputs = self.model( 2025-12-04T09:50:03.6478496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6478584Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6478823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6478907Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6479180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6479287Z return func(*args, **kwargs) 2025-12-04T09:50:03.6479562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6479652Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6479656Z 2025-12-04T09:50:03.6479764Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6479986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6480056Z res = mod(**inputs) 2025-12-04T09:50:03.6480324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6480406Z outputs = self.model( 2025-12-04T09:50:03.6480771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6480871Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6481115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6481202Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6481485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6481561Z return func(*args, **kwargs) 2025-12-04T09:50:03.6481837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:50:03.6481945Z hidden_states = residual + hidden_states 2025-12-04T09:50:03.6481949Z 2025-12-04T09:50:03.6482061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6482280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6482350Z res = mod(**inputs) 2025-12-04T09:50:03.6482616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6482700Z outputs = self.model( 2025-12-04T09:50:03.6482963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6483047Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6483267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6483349Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6483603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6483673Z return func(*args, **kwargs) 2025-12-04T09:50:03.6483926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6484028Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6484271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6484348Z return func(*args, **kwargs) 2025-12-04T09:50:03.6484593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6484707Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6484711Z 2025-12-04T09:50:03.6484822Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6485018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6485090Z res = mod(**inputs) 2025-12-04T09:50:03.6485340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6485409Z outputs = self.model( 2025-12-04T09:50:03.6485712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6485791Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6486016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6486102Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6486345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6486424Z return func(*args, **kwargs) 2025-12-04T09:50:03.6486672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6486773Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6487041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6487113Z return func(*args, **kwargs) 2025-12-04T09:50:03.6487367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6487449Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6487454Z 2025-12-04T09:50:03.6487556Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6487763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6487848Z res = mod(**inputs) 2025-12-04T09:50:03.6488104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6488181Z outputs = self.model( 2025-12-04T09:50:03.6488439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6488519Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6488755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6488835Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6489097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6489165Z return func(*args, **kwargs) 2025-12-04T09:50:03.6489429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6489537Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6489788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6489860Z return func(*args, **kwargs) 2025-12-04T09:50:03.6490136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6490259Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6490262Z 2025-12-04T09:50:03.6490379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6490593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6490668Z res = mod(**inputs) 2025-12-04T09:50:03.6490940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6491014Z outputs = self.model( 2025-12-04T09:50:03.6491303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6491376Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6491609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6491697Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6491975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6492055Z return func(*args, **kwargs) 2025-12-04T09:50:03.6492306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6492406Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6492659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6492729Z return func(*args, **kwargs) 2025-12-04T09:50:03.6492985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6493122Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6493140Z 2025-12-04T09:50:03.6493246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6493457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6493523Z res = mod(**inputs) 2025-12-04T09:50:03.6493775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6493853Z outputs = self.model( 2025-12-04T09:50:03.6494101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6494206Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6494453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6494540Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6494816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6494891Z return func(*args, **kwargs) 2025-12-04T09:50:03.6495179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6495285Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6495559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6495636Z return func(*args, **kwargs) 2025-12-04T09:50:03.6495895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6495987Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6495990Z 2025-12-04T09:50:03.6496105Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6496312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6496384Z res = mod(**inputs) 2025-12-04T09:50:03.6496650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6496719Z outputs = self.model( 2025-12-04T09:50:03.6496995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6497070Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6497304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6497395Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6497652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6497733Z return func(*args, **kwargs) 2025-12-04T09:50:03.6498021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6498163Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6498446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6498519Z return func(*args, **kwargs) 2025-12-04T09:50:03.6498786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6498888Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6498893Z 2025-12-04T09:50:03.6499003Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6499224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6499289Z res = mod(**inputs) 2025-12-04T09:50:03.6499569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6499650Z outputs = self.model( 2025-12-04T09:50:03.6499900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6499982Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6500207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6500287Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6500536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6500622Z return func(*args, **kwargs) 2025-12-04T09:50:03.6500876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6500990Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6501263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6501346Z return func(*args, **kwargs) 2025-12-04T09:50:03.6501619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6501750Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6501753Z 2025-12-04T09:50:03.6501864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6502064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6502138Z res = mod(**inputs) 2025-12-04T09:50:03.6502393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6502461Z outputs = self.model( 2025-12-04T09:50:03.6502778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6502860Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6503113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6503205Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6503471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6503547Z return func(*args, **kwargs) 2025-12-04T09:50:03.6503803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6503903Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6504160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6504229Z return func(*args, **kwargs) 2025-12-04T09:50:03.6504491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6504610Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6504614Z 2025-12-04T09:50:03.6504717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6504923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6504987Z res = mod(**inputs) 2025-12-04T09:50:03.6505235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6505311Z outputs = self.model( 2025-12-04T09:50:03.6505560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6505641Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6505882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6505970Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6506238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6506311Z return func(*args, **kwargs) 2025-12-04T09:50:03.6506582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6506714Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6506717Z 2025-12-04T09:50:03.6506840Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6507050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6507118Z res = mod(**inputs) 2025-12-04T09:50:03.6507381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6507464Z outputs = self.model( 2025-12-04T09:50:03.6507734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6507820Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6508056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6508141Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6508404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6508480Z return func(*args, **kwargs) 2025-12-04T09:50:03.6508743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6508879Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6509112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6509195Z return self.act(input) 2025-12-04T09:50:03.6509200Z 2025-12-04T09:50:03.6509311Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6509521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6509597Z res = mod(**inputs) 2025-12-04T09:50:03.6509860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6509940Z outputs = self.model( 2025-12-04T09:50:03.6510204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6510281Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6510524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6510609Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6510917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6511001Z return func(*args, **kwargs) 2025-12-04T09:50:03.6511271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6511367Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6511371Z 2025-12-04T09:50:03.6511481Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6511699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6511777Z res = mod(**inputs) 2025-12-04T09:50:03.6512047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6512138Z outputs = self.model( 2025-12-04T09:50:03.6512432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6512517Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6512760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6512846Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6513103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6513183Z return func(*args, **kwargs) 2025-12-04T09:50:03.6513468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6513579Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6513837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6513912Z return func(*args, **kwargs) 2025-12-04T09:50:03.6514192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6514311Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6514315Z 2025-12-04T09:50:03.6514423Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6514641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6514710Z res = mod(**inputs) 2025-12-04T09:50:03.6514984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6515058Z outputs = self.model( 2025-12-04T09:50:03.6515323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6515411Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6515649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6515745Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6516004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6516077Z return func(*args, **kwargs) 2025-12-04T09:50:03.6516350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6516456Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6516716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6516798Z return func(*args, **kwargs) 2025-12-04T09:50:03.6517064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6517158Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6517180Z 2025-12-04T09:50:03.6517308Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6517520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6517597Z res = mod(**inputs) 2025-12-04T09:50:03.6517860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6517932Z outputs = self.model( 2025-12-04T09:50:03.6518204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6518285Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6518528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6518611Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6518883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6518969Z return func(*args, **kwargs) 2025-12-04T09:50:03.6519232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6519344Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6519602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6519694Z return func(*args, **kwargs) 2025-12-04T09:50:03.6519968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6520085Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6520088Z 2025-12-04T09:50:03.6520195Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6520413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6520723Z res = mod(**inputs) 2025-12-04T09:50:03.6521008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6521081Z outputs = self.model( 2025-12-04T09:50:03.6521350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6521442Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6521687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6521786Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6522053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6522129Z return func(*args, **kwargs) 2025-12-04T09:50:03.6522401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6522509Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6522765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6522846Z return func(*args, **kwargs) 2025-12-04T09:50:03.6523111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6523270Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6523274Z 2025-12-04T09:50:03.6523379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6523577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6523651Z res = mod(**inputs) 2025-12-04T09:50:03.6523902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6524014Z outputs = self.model( 2025-12-04T09:50:03.6524270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6524345Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6524578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6524658Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6524910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6524992Z return func(*args, **kwargs) 2025-12-04T09:50:03.6525258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6525387Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6525653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6525727Z return func(*args, **kwargs) 2025-12-04T09:50:03.6525997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6526091Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6526095Z 2025-12-04T09:50:03.6526204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6526442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6526512Z res = mod(**inputs) 2025-12-04T09:50:03.6526784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6526858Z outputs = self.model( 2025-12-04T09:50:03.6527123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6527212Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6527451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6527536Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6527800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6527872Z return func(*args, **kwargs) 2025-12-04T09:50:03.6528145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6528249Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6528510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6528591Z return func(*args, **kwargs) 2025-12-04T09:50:03.6528861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6528971Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6528974Z 2025-12-04T09:50:03.6529082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6529293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6529369Z res = mod(**inputs) 2025-12-04T09:50:03.6529636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6529708Z outputs = self.model( 2025-12-04T09:50:03.6529979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6530059Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6530323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6530455Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6530714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6530792Z return func(*args, **kwargs) 2025-12-04T09:50:03.6531048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6531155Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6531403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6531473Z return func(*args, **kwargs) 2025-12-04T09:50:03.6531743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6531875Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6531880Z 2025-12-04T09:50:03.6531983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6532187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6532252Z res = mod(**inputs) 2025-12-04T09:50:03.6532518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6532592Z outputs = self.model( 2025-12-04T09:50:03.6532885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6532972Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6533208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6533301Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6533563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6533635Z return func(*args, **kwargs) 2025-12-04T09:50:03.6533908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6534011Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6534268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6534350Z return func(*args, **kwargs) 2025-12-04T09:50:03.6534614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6534706Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6534709Z 2025-12-04T09:50:03.6534827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6535029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6535105Z res = mod(**inputs) 2025-12-04T09:50:03.6535367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6535439Z outputs = self.model( 2025-12-04T09:50:03.6535708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6535786Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6536032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6536113Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6536369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6536449Z return func(*args, **kwargs) 2025-12-04T09:50:03.6536758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6536895Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6536898Z 2025-12-04T09:50:03.6537008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6537218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6537291Z res = mod(**inputs) 2025-12-04T09:50:03.6537555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6537628Z outputs = self.model( 2025-12-04T09:50:03.6537899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6537978Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6538245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6538334Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6538591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6538671Z return func(*args, **kwargs) 2025-12-04T09:50:03.6538943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6539069Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6539323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6539398Z return self.act(input) 2025-12-04T09:50:03.6539402Z 2025-12-04T09:50:03.6539520Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6539740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6539810Z res = mod(**inputs) 2025-12-04T09:50:03.6540081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6540152Z outputs = self.model( 2025-12-04T09:50:03.6540434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6540513Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6540760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6540855Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6541112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6541185Z return func(*args, **kwargs) 2025-12-04T09:50:03.6541466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6541558Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6541562Z 2025-12-04T09:50:03.6541678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6541897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6541966Z res = mod(**inputs) 2025-12-04T09:50:03.6542237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6542312Z outputs = self.model( 2025-12-04T09:50:03.6542593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6542672Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6542917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6543027Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6543309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6543384Z return func(*args, **kwargs) 2025-12-04T09:50:03.6543669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:50:03.6543756Z hidden_states = residual + hidden_states 2025-12-04T09:50:03.6543759Z 2025-12-04T09:50:03.6543876Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6544086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6544154Z res = mod(**inputs) 2025-12-04T09:50:03.6544430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6544520Z outputs = self.model( 2025-12-04T09:50:03.6544808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6544895Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6545141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6545233Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6545502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6545604Z return func(*args, **kwargs) 2025-12-04T09:50:03.6545877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6545984Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6546259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6546335Z return func(*args, **kwargs) 2025-12-04T09:50:03.6546598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6546727Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6546730Z 2025-12-04T09:50:03.6546839Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6547215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6547303Z res = mod(**inputs) 2025-12-04T09:50:03.6547576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6547657Z outputs = self.model( 2025-12-04T09:50:03.6547937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6548018Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6548269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6548355Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6548628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6548710Z return func(*args, **kwargs) 2025-12-04T09:50:03.6548974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6549092Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6549354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6549427Z return func(*args, **kwargs) 2025-12-04T09:50:03.6549703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6549832Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6549861Z 2025-12-04T09:50:03.6549980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6550188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6550255Z res = mod(**inputs) 2025-12-04T09:50:03.6550524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6550596Z outputs = self.model( 2025-12-04T09:50:03.6550858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6550943Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6551178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6551295Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6551558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6551632Z return func(*args, **kwargs) 2025-12-04T09:50:03.6551900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6552025Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6552291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6552388Z return func(*args, **kwargs) 2025-12-04T09:50:03.6552654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6552780Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6552784Z 2025-12-04T09:50:03.6552896Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6553110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6553189Z res = mod(**inputs) 2025-12-04T09:50:03.6553458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6553539Z outputs = self.model( 2025-12-04T09:50:03.6553803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6553885Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6554133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6554217Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6554490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6554568Z return func(*args, **kwargs) 2025-12-04T09:50:03.6554822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6554928Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6555174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6555245Z return func(*args, **kwargs) 2025-12-04T09:50:03.6555507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6555645Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6555648Z 2025-12-04T09:50:03.6555760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6555960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6556027Z res = mod(**inputs) 2025-12-04T09:50:03.6556317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6556387Z outputs = self.model( 2025-12-04T09:50:03.6556636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6556719Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6556943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6557032Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6557275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6557344Z return func(*args, **kwargs) 2025-12-04T09:50:03.6557637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6557746Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6558011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6558084Z return func(*args, **kwargs) 2025-12-04T09:50:03.6558345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6558445Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6558449Z 2025-12-04T09:50:03.6558580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6558790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6558866Z res = mod(**inputs) 2025-12-04T09:50:03.6559132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6559214Z outputs = self.model( 2025-12-04T09:50:03.6559484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6559563Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6559816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6559900Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6560161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6560243Z return func(*args, **kwargs) 2025-12-04T09:50:03.6560559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6560680Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6560942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6561017Z return func(*args, **kwargs) 2025-12-04T09:50:03.6561290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6561390Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6561394Z 2025-12-04T09:50:03.6561509Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6561719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6561789Z res = mod(**inputs) 2025-12-04T09:50:03.6562065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6562135Z outputs = self.model( 2025-12-04T09:50:03.6562382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6562465Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6562753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6562842Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6563088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6563159Z return func(*args, **kwargs) 2025-12-04T09:50:03.6563416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6563517Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6563767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6563835Z return func(*args, **kwargs) 2025-12-04T09:50:03.6564100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6564242Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6564245Z 2025-12-04T09:50:03.6564347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6564544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6564614Z res = mod(**inputs) 2025-12-04T09:50:03.6564865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6564958Z outputs = self.model( 2025-12-04T09:50:03.6565210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6565284Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6565523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6565602Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6565852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6565928Z return func(*args, **kwargs) 2025-12-04T09:50:03.6566181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6566285Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6566531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6566602Z return func(*args, **kwargs) 2025-12-04T09:50:03.6566860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6566942Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6566945Z 2025-12-04T09:50:03.6567056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6567258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6567323Z res = mod(**inputs) 2025-12-04T09:50:03.6567580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6567650Z outputs = self.model( 2025-12-04T09:50:03.6567900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6567982Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6568210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6568295Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6568541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6568630Z return func(*args, **kwargs) 2025-12-04T09:50:03.6568904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6569026Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6569030Z 2025-12-04T09:50:03.6569140Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6569338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6569402Z res = mod(**inputs) 2025-12-04T09:50:03.6569659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6569726Z outputs = self.model( 2025-12-04T09:50:03.6569974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6570072Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6570304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6570392Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6570637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6570706Z return func(*args, **kwargs) 2025-12-04T09:50:03.6570963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6571100Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6571318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6571397Z return self.act(input) 2025-12-04T09:50:03.6571404Z 2025-12-04T09:50:03.6571511Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6571724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6571795Z res = mod(**inputs) 2025-12-04T09:50:03.6572049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6572129Z outputs = self.model( 2025-12-04T09:50:03.6572383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6572469Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6572698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6572779Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6573048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6573126Z return func(*args, **kwargs) 2025-12-04T09:50:03.6573396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6573502Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6573506Z 2025-12-04T09:50:03.6573613Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6573822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6573889Z res = mod(**inputs) 2025-12-04T09:50:03.6574149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6574235Z outputs = self.model( 2025-12-04T09:50:03.6574504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6574583Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6574833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6574955Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6575222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6575296Z return func(*args, **kwargs) 2025-12-04T09:50:03.6575557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6575670Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6575926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6576007Z return func(*args, **kwargs) 2025-12-04T09:50:03.6576269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6576409Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6576414Z 2025-12-04T09:50:03.6576534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6576746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6576814Z res = mod(**inputs) 2025-12-04T09:50:03.6577085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6577157Z outputs = self.model( 2025-12-04T09:50:03.6577425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6577526Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6577763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6577855Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6578116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6578200Z return func(*args, **kwargs) 2025-12-04T09:50:03.6578465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6578570Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6578833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6578906Z return func(*args, **kwargs) 2025-12-04T09:50:03.6579170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6579264Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6579268Z 2025-12-04T09:50:03.6579378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6579593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6579665Z res = mod(**inputs) 2025-12-04T09:50:03.6579928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6580011Z outputs = self.model( 2025-12-04T09:50:03.6580283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6580361Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6580613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6580696Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6580958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6581031Z return func(*args, **kwargs) 2025-12-04T09:50:03.6581305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6581462Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6581723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6581801Z return func(*args, **kwargs) 2025-12-04T09:50:03.6582063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6582182Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6582187Z 2025-12-04T09:50:03.6582367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6582592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6582659Z res = mod(**inputs) 2025-12-04T09:50:03.6582952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6583030Z outputs = self.model( 2025-12-04T09:50:03.6583318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6583393Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6583622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6583710Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6583958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6584055Z return func(*args, **kwargs) 2025-12-04T09:50:03.6584306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6584406Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6584659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6584727Z return func(*args, **kwargs) 2025-12-04T09:50:03.6584976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6585120Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6585124Z 2025-12-04T09:50:03.6585227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6585433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6585498Z res = mod(**inputs) 2025-12-04T09:50:03.6585748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6585823Z outputs = self.model( 2025-12-04T09:50:03.6586074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6586150Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6586381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6586460Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6586714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6586783Z return func(*args, **kwargs) 2025-12-04T09:50:03.6587035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6587143Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6587390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6587468Z return func(*args, **kwargs) 2025-12-04T09:50:03.6587770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6587861Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6587865Z 2025-12-04T09:50:03.6587977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6588176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6588242Z res = mod(**inputs) 2025-12-04T09:50:03.6588502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6588573Z outputs = self.model( 2025-12-04T09:50:03.6588829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6588904Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6589143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6589234Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6589479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6589557Z return func(*args, **kwargs) 2025-12-04T09:50:03.6589805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6589903Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6590170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6590238Z return func(*args, **kwargs) 2025-12-04T09:50:03.6590488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6590594Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6590599Z 2025-12-04T09:50:03.6590703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6590905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6590971Z res = mod(**inputs) 2025-12-04T09:50:03.6591220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6591297Z outputs = self.model( 2025-12-04T09:50:03.6591543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6591619Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6591849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6591927Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6592178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6592250Z return func(*args, **kwargs) 2025-12-04T09:50:03.6592498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6592606Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6592848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6592928Z return func(*args, **kwargs) 2025-12-04T09:50:03.6593177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6593308Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6593311Z 2025-12-04T09:50:03.6593423Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6593620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6593723Z res = mod(**inputs) 2025-12-04T09:50:03.6593982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6594050Z outputs = self.model( 2025-12-04T09:50:03.6594309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6594385Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6594609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6594697Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6594941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6595035Z return func(*args, **kwargs) 2025-12-04T09:50:03.6595286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6595385Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6595633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6595702Z return func(*args, **kwargs) 2025-12-04T09:50:03.6595948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6596056Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6596059Z 2025-12-04T09:50:03.6596163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6596369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6596434Z res = mod(**inputs) 2025-12-04T09:50:03.6596684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6596764Z outputs = self.model( 2025-12-04T09:50:03.6597024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6597102Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6597342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6597425Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6597689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6597761Z return func(*args, **kwargs) 2025-12-04T09:50:03.6598022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6598158Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6598163Z 2025-12-04T09:50:03.6598272Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6598486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6598554Z res = mod(**inputs) 2025-12-04T09:50:03.6598825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6598906Z outputs = self.model( 2025-12-04T09:50:03.6599170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6599248Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6599491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6599572Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6599839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6599951Z return func(*args, **kwargs) 2025-12-04T09:50:03.6600218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6600352Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6600665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6600754Z return self.act(input) 2025-12-04T09:50:03.6600761Z 2025-12-04T09:50:03.6600873Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6601093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6601170Z res = mod(**inputs) 2025-12-04T09:50:03.6601475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6601553Z outputs = self.model( 2025-12-04T09:50:03.6601822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6601901Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6602159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6602244Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6602514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6602616Z return func(*args, **kwargs) 2025-12-04T09:50:03.6602884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6602972Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6602985Z 2025-12-04T09:50:03.6603098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6603314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6603392Z res = mod(**inputs) 2025-12-04T09:50:03.6603663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6603735Z outputs = self.model( 2025-12-04T09:50:03.6604012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6604093Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6604349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6604433Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6604705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6604789Z return func(*args, **kwargs) 2025-12-04T09:50:03.6605074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:50:03.6605160Z hidden_states = residual + hidden_states 2025-12-04T09:50:03.6605163Z 2025-12-04T09:50:03.6605278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6605488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6605561Z res = mod(**inputs) 2025-12-04T09:50:03.6605846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6605918Z outputs = self.model( 2025-12-04T09:50:03.6606198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6606277Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6606545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6606654Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6606920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6606999Z return func(*args, **kwargs) 2025-12-04T09:50:03.6607273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6607380Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6607652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6607724Z return func(*args, **kwargs) 2025-12-04T09:50:03.6608022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6608146Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6608152Z 2025-12-04T09:50:03.6608260Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6608479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6608548Z res = mod(**inputs) 2025-12-04T09:50:03.6608811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6608887Z outputs = self.model( 2025-12-04T09:50:03.6609154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6609236Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6609462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6609544Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6609796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6609868Z return func(*args, **kwargs) 2025-12-04T09:50:03.6610130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6610226Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6610464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6610540Z return func(*args, **kwargs) 2025-12-04T09:50:03.6610789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6610870Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6610874Z 2025-12-04T09:50:03.6610986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6611189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6611259Z res = mod(**inputs) 2025-12-04T09:50:03.6611507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6611576Z outputs = self.model( 2025-12-04T09:50:03.6611830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6611904Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6612128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6612214Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6612464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6612539Z return func(*args, **kwargs) 2025-12-04T09:50:03.6612813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6612911Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6613153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6613218Z return func(*args, **kwargs) 2025-12-04T09:50:03.6613466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6613575Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6613578Z 2025-12-04T09:50:03.6613677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6613877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6613955Z res = mod(**inputs) 2025-12-04T09:50:03.6614202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6614279Z outputs = self.model( 2025-12-04T09:50:03.6614529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6614612Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6614838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6614922Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6615191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6615260Z return func(*args, **kwargs) 2025-12-04T09:50:03.6615508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6615615Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6615860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6615936Z return func(*args, **kwargs) 2025-12-04T09:50:03.6616186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6616323Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6616326Z 2025-12-04T09:50:03.6616436Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6616633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6616707Z res = mod(**inputs) 2025-12-04T09:50:03.6616956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6617025Z outputs = self.model( 2025-12-04T09:50:03.6617285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6617360Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6617583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6617670Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6617924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6618008Z return func(*args, **kwargs) 2025-12-04T09:50:03.6618270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6618374Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6618637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6618730Z return func(*args, **kwargs) 2025-12-04T09:50:03.6619030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6619122Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6619125Z 2025-12-04T09:50:03.6619230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6619435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6619499Z res = mod(**inputs) 2025-12-04T09:50:03.6619749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6619827Z outputs = self.model( 2025-12-04T09:50:03.6620076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6620171Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6620396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6620476Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6620735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6620804Z return func(*args, **kwargs) 2025-12-04T09:50:03.6621071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6621194Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6621461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6621541Z return func(*args, **kwargs) 2025-12-04T09:50:03.6621813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6621916Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6621921Z 2025-12-04T09:50:03.6622036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6622246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6622319Z res = mod(**inputs) 2025-12-04T09:50:03.6622571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6622639Z outputs = self.model( 2025-12-04T09:50:03.6622895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6622969Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6623193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6623283Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6623541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6623621Z return func(*args, **kwargs) 2025-12-04T09:50:03.6623894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6623998Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6624279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6624353Z return func(*args, **kwargs) 2025-12-04T09:50:03.6624634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6624771Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6624774Z 2025-12-04T09:50:03.6624884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6625138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6625210Z res = mod(**inputs) 2025-12-04T09:50:03.6625486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6625567Z outputs = self.model( 2025-12-04T09:50:03.6625884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6625974Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6626216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6626300Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6626580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6626656Z return func(*args, **kwargs) 2025-12-04T09:50:03.6626936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6627042Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6627305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6627384Z return func(*args, **kwargs) 2025-12-04T09:50:03.6627657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6627775Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6627778Z 2025-12-04T09:50:03.6627898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6628109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6628186Z res = mod(**inputs) 2025-12-04T09:50:03.6628475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6628547Z outputs = self.model( 2025-12-04T09:50:03.6628824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6628901Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6629144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6629235Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6629503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6629582Z return func(*args, **kwargs) 2025-12-04T09:50:03.6629859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6629987Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6629992Z 2025-12-04T09:50:03.6630108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6630318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6630393Z res = mod(**inputs) 2025-12-04T09:50:03.6630668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6630741Z outputs = self.model( 2025-12-04T09:50:03.6631028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6631106Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6631353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6631445Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6631763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6631846Z return func(*args, **kwargs) 2025-12-04T09:50:03.6632113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6632239Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6632480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6632557Z return self.act(input) 2025-12-04T09:50:03.6632561Z 2025-12-04T09:50:03.6632678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6632889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6632957Z res = mod(**inputs) 2025-12-04T09:50:03.6633250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6633328Z outputs = self.model( 2025-12-04T09:50:03.6633594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6633681Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6633920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6634014Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6634290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6634365Z return func(*args, **kwargs) 2025-12-04T09:50:03.6634632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6634719Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6634724Z 2025-12-04T09:50:03.6634835Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6635049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6635118Z res = mod(**inputs) 2025-12-04T09:50:03.6635385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6635459Z outputs = self.model( 2025-12-04T09:50:03.6635723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6635811Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6636046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6636136Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6636396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6636474Z return func(*args, **kwargs) 2025-12-04T09:50:03.6636742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6636846Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6637102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6637185Z return func(*args, **kwargs) 2025-12-04T09:50:03.6637451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6637582Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6637585Z 2025-12-04T09:50:03.6637686Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6637885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6637990Z res = mod(**inputs) 2025-12-04T09:50:03.6638261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6638334Z outputs = self.model( 2025-12-04T09:50:03.6638590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6638664Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6638907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6638994Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6639252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6639333Z return func(*args, **kwargs) 2025-12-04T09:50:03.6639611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6639727Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6639986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6640058Z return func(*args, **kwargs) 2025-12-04T09:50:03.6640330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6640418Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6640439Z 2025-12-04T09:50:03.6640624Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6640859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6640929Z res = mod(**inputs) 2025-12-04T09:50:03.6641216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6641293Z outputs = self.model( 2025-12-04T09:50:03.6641566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6641657Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6641903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6641998Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6642258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6642330Z return func(*args, **kwargs) 2025-12-04T09:50:03.6642585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6642686Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6642929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6643011Z return func(*args, **kwargs) 2025-12-04T09:50:03.6643261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6643382Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6643386Z 2025-12-04T09:50:03.6643490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6643688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6643762Z res = mod(**inputs) 2025-12-04T09:50:03.6644011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6644080Z outputs = self.model( 2025-12-04T09:50:03.6644335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6644434Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6644686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6644768Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6645009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6645084Z return func(*args, **kwargs) 2025-12-04T09:50:03.6645330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6645434Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6645675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6645758Z return func(*args, **kwargs) 2025-12-04T09:50:03.6646018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6646155Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6646159Z 2025-12-04T09:50:03.6646263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6646471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6646537Z res = mod(**inputs) 2025-12-04T09:50:03.6646792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6646877Z outputs = self.model( 2025-12-04T09:50:03.6647232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6647320Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6647550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6647641Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6647885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6647954Z return func(*args, **kwargs) 2025-12-04T09:50:03.6648213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6648312Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6648556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6648635Z return func(*args, **kwargs) 2025-12-04T09:50:03.6648885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6648981Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6648986Z 2025-12-04T09:50:03.6649093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6649290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6649367Z res = mod(**inputs) 2025-12-04T09:50:03.6649616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6649688Z outputs = self.model( 2025-12-04T09:50:03.6649941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6650017Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6650247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6650328Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6650572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6650716Z return func(*args, **kwargs) 2025-12-04T09:50:03.6650966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6651071Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6651311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6651379Z return func(*args, **kwargs) 2025-12-04T09:50:03.6651633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6651730Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6651735Z 2025-12-04T09:50:03.6651835Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6652079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6652150Z res = mod(**inputs) 2025-12-04T09:50:03.6652414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6652483Z outputs = self.model( 2025-12-04T09:50:03.6652730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6652812Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6653035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6653148Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6653391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6653462Z return func(*args, **kwargs) 2025-12-04T09:50:03.6653714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6653815Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6654056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6654133Z return func(*args, **kwargs) 2025-12-04T09:50:03.6654379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6654515Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6654520Z 2025-12-04T09:50:03.6654624Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6654820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6654894Z res = mod(**inputs) 2025-12-04T09:50:03.6655143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6655216Z outputs = self.model( 2025-12-04T09:50:03.6655473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6655547Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6655777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6655856Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6656100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6656177Z return func(*args, **kwargs) 2025-12-04T09:50:03.6656425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6656532Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6656809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6656881Z return func(*args, **kwargs) 2025-12-04T09:50:03.6657141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6657223Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6657227Z 2025-12-04T09:50:03.6657331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6657537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6657603Z res = mod(**inputs) 2025-12-04T09:50:03.6657867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6657934Z outputs = self.model( 2025-12-04T09:50:03.6658195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6658280Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6658508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6658597Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6658845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6658919Z return func(*args, **kwargs) 2025-12-04T09:50:03.6659212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6659338Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6659342Z 2025-12-04T09:50:03.6659451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6659670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6659740Z res = mod(**inputs) 2025-12-04T09:50:03.6660013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6660084Z outputs = self.model( 2025-12-04T09:50:03.6660348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6660429Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6660652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6660734Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6660990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6661058Z return func(*args, **kwargs) 2025-12-04T09:50:03.6661308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6661427Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6661639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6661714Z return self.act(input) 2025-12-04T09:50:03.6661718Z 2025-12-04T09:50:03.6661820Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6662021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6662086Z res = mod(**inputs) 2025-12-04T09:50:03.6662331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6662405Z outputs = self.model( 2025-12-04T09:50:03.6662649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6662743Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6662984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6663064Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6663310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6663383Z return func(*args, **kwargs) 2025-12-04T09:50:03.6663651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6663742Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6663746Z 2025-12-04T09:50:03.6663848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6664075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6664154Z res = mod(**inputs) 2025-12-04T09:50:03.6664411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6664488Z outputs = self.model( 2025-12-04T09:50:03.6664759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6664832Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6665082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6665182Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6665448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6665519Z return func(*args, **kwargs) 2025-12-04T09:50:03.6665786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:50:03.6665874Z hidden_states = residual + hidden_states 2025-12-04T09:50:03.6665879Z 2025-12-04T09:50:03.6665983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6666179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6666251Z res = mod(**inputs) 2025-12-04T09:50:03.6666518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6666594Z outputs = self.model( 2025-12-04T09:50:03.6666862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6666937Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6667173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6667251Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6667497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6667574Z return func(*args, **kwargs) 2025-12-04T09:50:03.6667842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6667948Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6668206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6668279Z return func(*args, **kwargs) 2025-12-04T09:50:03.6668555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6668666Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6668670Z 2025-12-04T09:50:03.6668781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6668981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6669080Z res = mod(**inputs) 2025-12-04T09:50:03.6669356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6669425Z outputs = self.model( 2025-12-04T09:50:03.6669694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6669775Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6670001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6670089Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6670351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6670436Z return func(*args, **kwargs) 2025-12-04T09:50:03.6670700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6670800Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6671051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6671121Z return func(*args, **kwargs) 2025-12-04T09:50:03.6671392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6671511Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6671515Z 2025-12-04T09:50:03.6671615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6671807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6671877Z res = mod(**inputs) 2025-12-04T09:50:03.6672122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6672199Z outputs = self.model( 2025-12-04T09:50:03.6672441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6672513Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6672740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6672816Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6673056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6673130Z return func(*args, **kwargs) 2025-12-04T09:50:03.6673373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6673479Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6673717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6673784Z return func(*args, **kwargs) 2025-12-04T09:50:03.6674034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6674142Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6674145Z 2025-12-04T09:50:03.6674251Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6674445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6674507Z res = mod(**inputs) 2025-12-04T09:50:03.6674755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6674822Z outputs = self.model( 2025-12-04T09:50:03.6675065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6675186Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6675404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6675485Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6675719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6675787Z return func(*args, **kwargs) 2025-12-04T09:50:03.6676036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6676130Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6676392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6676462Z return func(*args, **kwargs) 2025-12-04T09:50:03.6676713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6676852Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6676855Z 2025-12-04T09:50:03.6676955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6677150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6677221Z res = mod(**inputs) 2025-12-04T09:50:03.6677480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6677554Z outputs = self.model( 2025-12-04T09:50:03.6677793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6677866Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6678093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6678171Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6678404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6678479Z return func(*args, **kwargs) 2025-12-04T09:50:03.6678726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6678835Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6679079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6679149Z return func(*args, **kwargs) 2025-12-04T09:50:03.6679409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6679499Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6679505Z 2025-12-04T09:50:03.6679615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6679814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6679878Z res = mod(**inputs) 2025-12-04T09:50:03.6680134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6680202Z outputs = self.model( 2025-12-04T09:50:03.6680453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6680611Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6680839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6680932Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6681219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6681291Z return func(*args, **kwargs) 2025-12-04T09:50:03.6681558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6681659Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6681923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6681999Z return func(*args, **kwargs) 2025-12-04T09:50:03.6682262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6682374Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6682378Z 2025-12-04T09:50:03.6682502Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6682722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6682799Z res = mod(**inputs) 2025-12-04T09:50:03.6683063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6683143Z outputs = self.model( 2025-12-04T09:50:03.6683408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6683486Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6683763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6683840Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6684083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6684161Z return func(*args, **kwargs) 2025-12-04T09:50:03.6684412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6684516Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6684758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6684828Z return func(*args, **kwargs) 2025-12-04T09:50:03.6685083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6685215Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6685218Z 2025-12-04T09:50:03.6685326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6685525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6685591Z res = mod(**inputs) 2025-12-04T09:50:03.6685846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6685917Z outputs = self.model( 2025-12-04T09:50:03.6686165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6686248Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6686469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6686555Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6686798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6686867Z return func(*args, **kwargs) 2025-12-04T09:50:03.6687126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6687223Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6687504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6687576Z return func(*args, **kwargs) 2025-12-04T09:50:03.6687824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6687913Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6687916Z 2025-12-04T09:50:03.6688018Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6688220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6688291Z res = mod(**inputs) 2025-12-04T09:50:03.6688538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6688630Z outputs = self.model( 2025-12-04T09:50:03.6688897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6688977Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6689222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6689303Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6689562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6689661Z return func(*args, **kwargs) 2025-12-04T09:50:03.6689922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6690056Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6690060Z 2025-12-04T09:50:03.6690171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6690377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6690456Z res = mod(**inputs) 2025-12-04T09:50:03.6690719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6690798Z outputs = self.model( 2025-12-04T09:50:03.6691060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6691138Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6691383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6691465Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6691722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6691805Z return func(*args, **kwargs) 2025-12-04T09:50:03.6692071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6692198Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6692413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6692487Z return self.act(input) 2025-12-04T09:50:03.6692491Z 2025-12-04T09:50:03.6692606Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6692813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6692891Z res = mod(**inputs) 2025-12-04T09:50:03.6693157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6693231Z outputs = self.model( 2025-12-04T09:50:03.6693502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6693616Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6693858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6693951Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6694214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6694307Z return func(*args, **kwargs) 2025-12-04T09:50:03.6694562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6694647Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6694651Z 2025-12-04T09:50:03.6694764Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6694982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6695054Z res = mod(**inputs) 2025-12-04T09:50:03.6695326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6695400Z outputs = self.model( 2025-12-04T09:50:03.6695676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6695752Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6695975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6696083Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6696328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6696408Z return func(*args, **kwargs) 2025-12-04T09:50:03.6696658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6696761Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6697013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6697082Z return func(*args, **kwargs) 2025-12-04T09:50:03.6697331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6697451Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6697456Z 2025-12-04T09:50:03.6697560Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6697765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6697829Z res = mod(**inputs) 2025-12-04T09:50:03.6698083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6698162Z outputs = self.model( 2025-12-04T09:50:03.6698410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6698494Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6698731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6698816Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6699082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6699155Z return func(*args, **kwargs) 2025-12-04T09:50:03.6699420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6699534Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6699796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6699925Z return func(*args, **kwargs) 2025-12-04T09:50:03.6700190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6700277Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6700281Z 2025-12-04T09:50:03.6700398Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6700608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6700687Z res = mod(**inputs) 2025-12-04T09:50:03.6700940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6701009Z outputs = self.model( 2025-12-04T09:50:03.6701279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6701357Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6701581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6701665Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6701906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6701983Z return func(*args, **kwargs) 2025-12-04T09:50:03.6702228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6702348Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6702595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6702664Z return func(*args, **kwargs) 2025-12-04T09:50:03.6702915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6703038Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6703042Z 2025-12-04T09:50:03.6703144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6703348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6703413Z res = mod(**inputs) 2025-12-04T09:50:03.6703665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6703742Z outputs = self.model( 2025-12-04T09:50:03.6703990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6704064Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6704295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6704377Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6704632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6704702Z return func(*args, **kwargs) 2025-12-04T09:50:03.6704950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6705057Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6705302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6705377Z return func(*args, **kwargs) 2025-12-04T09:50:03.6705628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6705767Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6705789Z 2025-12-04T09:50:03.6705901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6706116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6706184Z res = mod(**inputs) 2025-12-04T09:50:03.6706442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6706512Z outputs = self.model( 2025-12-04T09:50:03.6706769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6706846Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6707071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6707159Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6707420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6707501Z return func(*args, **kwargs) 2025-12-04T09:50:03.6707749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6707848Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6708109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6708178Z return func(*args, **kwargs) 2025-12-04T09:50:03.6708445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6708543Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6708547Z 2025-12-04T09:50:03.6708649Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6708857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6708924Z res = mod(**inputs) 2025-12-04T09:50:03.6709173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6709253Z outputs = self.model( 2025-12-04T09:50:03.6709500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6709574Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6709802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6709885Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6710152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6710225Z return func(*args, **kwargs) 2025-12-04T09:50:03.6710489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6710605Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6710860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6710941Z return func(*args, **kwargs) 2025-12-04T09:50:03.6711200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6711301Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6711306Z 2025-12-04T09:50:03.6711422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6711630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6711698Z res = mod(**inputs) 2025-12-04T09:50:03.6711975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6712067Z outputs = self.model( 2025-12-04T09:50:03.6712343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6712420Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6712643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6712730Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6712971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6713051Z return func(*args, **kwargs) 2025-12-04T09:50:03.6713311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6713414Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6713693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6713774Z return func(*args, **kwargs) 2025-12-04T09:50:03.6714040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6714185Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6714189Z 2025-12-04T09:50:03.6714296Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6714514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6714604Z res = mod(**inputs) 2025-12-04T09:50:03.6714873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6714955Z outputs = self.model( 2025-12-04T09:50:03.6715221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6715303Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6715547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6715631Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6715900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6715974Z return func(*args, **kwargs) 2025-12-04T09:50:03.6716240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6716352Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6716611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6716694Z return func(*args, **kwargs) 2025-12-04T09:50:03.6716964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6717055Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6717058Z 2025-12-04T09:50:03.6717176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6717385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6717453Z res = mod(**inputs) 2025-12-04T09:50:03.6717727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6717801Z outputs = self.model( 2025-12-04T09:50:03.6718071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6718148Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6718386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6718515Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6718771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6718852Z return func(*args, **kwargs) 2025-12-04T09:50:03.6719112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6719241Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6719246Z 2025-12-04T09:50:03.6719363Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6719573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6719640Z res = mod(**inputs) 2025-12-04T09:50:03.6719929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6720006Z outputs = self.model( 2025-12-04T09:50:03.6720284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6720364Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6720672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6720773Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6721037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6721149Z return func(*args, **kwargs) 2025-12-04T09:50:03.6721439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6721573Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6721827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6721907Z return self.act(input) 2025-12-04T09:50:03.6721913Z 2025-12-04T09:50:03.6722027Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6722262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6722333Z res = mod(**inputs) 2025-12-04T09:50:03.6722617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6722692Z outputs = self.model( 2025-12-04T09:50:03.6722969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6723057Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6723306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6723389Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6723671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6723746Z return func(*args, **kwargs) 2025-12-04T09:50:03.6724027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6724116Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6724121Z 2025-12-04T09:50:03.6724233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6724459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6724527Z res = mod(**inputs) 2025-12-04T09:50:03.6724806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6724880Z outputs = self.model( 2025-12-04T09:50:03.6725171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6725277Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6725516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6725600Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6725871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6725946Z return func(*args, **kwargs) 2025-12-04T09:50:03.6726220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:50:03.6726306Z hidden_states = residual + hidden_states 2025-12-04T09:50:03.6726310Z 2025-12-04T09:50:03.6726417Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6726655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6726729Z res = mod(**inputs) 2025-12-04T09:50:03.6726992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6727075Z outputs = self.model( 2025-12-04T09:50:03.6727338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6727425Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6727660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6727779Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6728044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6728119Z return func(*args, **kwargs) 2025-12-04T09:50:03.6728388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6728496Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6728752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6728833Z return func(*args, **kwargs) 2025-12-04T09:50:03.6729097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6729218Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6729223Z 2025-12-04T09:50:03.6729338Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6729548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6729625Z res = mod(**inputs) 2025-12-04T09:50:03.6729889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6729966Z outputs = self.model( 2025-12-04T09:50:03.6730233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6730312Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6730546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6730638Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6730901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6730981Z return func(*args, **kwargs) 2025-12-04T09:50:03.6731240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6731345Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6731649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6731725Z return func(*args, **kwargs) 2025-12-04T09:50:03.6731999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6732090Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6732094Z 2025-12-04T09:50:03.6732204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6732426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6732505Z res = mod(**inputs) 2025-12-04T09:50:03.6732757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6732834Z outputs = self.model( 2025-12-04T09:50:03.6733100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6733183Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6733406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6733484Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6733733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6733804Z return func(*args, **kwargs) 2025-12-04T09:50:03.6734081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6734182Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6734423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6734498Z return func(*args, **kwargs) 2025-12-04T09:50:03.6734750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6734862Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6734865Z 2025-12-04T09:50:03.6734976Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6735176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6735246Z res = mod(**inputs) 2025-12-04T09:50:03.6735496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6735565Z outputs = self.model( 2025-12-04T09:50:03.6735818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6735892Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6736118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6736208Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6736449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6736525Z return func(*args, **kwargs) 2025-12-04T09:50:03.6736772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6736871Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6737123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6737193Z return func(*args, **kwargs) 2025-12-04T09:50:03.6737448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6737588Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6737606Z 2025-12-04T09:50:03.6737726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6737932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6737997Z res = mod(**inputs) 2025-12-04T09:50:03.6738247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6738324Z outputs = self.model( 2025-12-04T09:50:03.6738571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6738652Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6738874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6738978Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6739230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6739300Z return func(*args, **kwargs) 2025-12-04T09:50:03.6739557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6739658Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6739901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6740000Z return func(*args, **kwargs) 2025-12-04T09:50:03.6740256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6740345Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6740348Z 2025-12-04T09:50:03.6740461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6740666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6740747Z res = mod(**inputs) 2025-12-04T09:50:03.6741016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6741089Z outputs = self.model( 2025-12-04T09:50:03.6741365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6741443Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6741686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6741777Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6742040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6742123Z return func(*args, **kwargs) 2025-12-04T09:50:03.6742395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6742503Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6742780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6742854Z return func(*args, **kwargs) 2025-12-04T09:50:03.6743130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6743233Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6743237Z 2025-12-04T09:50:03.6743346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6743568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6743636Z res = mod(**inputs) 2025-12-04T09:50:03.6743906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6744025Z outputs = self.model( 2025-12-04T09:50:03.6744292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6744378Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6744617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6744695Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6744949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6745016Z return func(*args, **kwargs) 2025-12-04T09:50:03.6745276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6745399Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6745644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6745720Z return func(*args, **kwargs) 2025-12-04T09:50:03.6745968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6746096Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6746100Z 2025-12-04T09:50:03.6746209Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6746427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6746497Z res = mod(**inputs) 2025-12-04T09:50:03.6746745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6746813Z outputs = self.model( 2025-12-04T09:50:03.6747241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6747325Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6747554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6747641Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6747888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6747966Z return func(*args, **kwargs) 2025-12-04T09:50:03.6748222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6748321Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6748575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6748646Z return func(*args, **kwargs) 2025-12-04T09:50:03.6748906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6748990Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6748994Z 2025-12-04T09:50:03.6749097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6749303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6749370Z res = mod(**inputs) 2025-12-04T09:50:03.6749622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6749700Z outputs = self.model( 2025-12-04T09:50:03.6749947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6750032Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6750292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6750394Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6767703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6767956Z return func(*args, **kwargs) 2025-12-04T09:50:03.6768294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6768428Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6768453Z 2025-12-04T09:50:03.6768580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6768797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6768871Z res = mod(**inputs) 2025-12-04T09:50:03.6769281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6769375Z outputs = self.model( 2025-12-04T09:50:03.6769640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6769722Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6769954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6770052Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6770357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6770442Z return func(*args, **kwargs) 2025-12-04T09:50:03.6770696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6770823Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6771057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6771133Z return self.act(input) 2025-12-04T09:50:03.6771137Z 2025-12-04T09:50:03.6771254Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6771469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6771539Z res = mod(**inputs) 2025-12-04T09:50:03.6771801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6771878Z outputs = self.model( 2025-12-04T09:50:03.6772129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6772217Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6772499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6772591Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6772849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6772921Z return func(*args, **kwargs) 2025-12-04T09:50:03.6773183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6773270Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6773274Z 2025-12-04T09:50:03.6773387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6773602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6773671Z res = mod(**inputs) 2025-12-04T09:50:03.6773930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6774003Z outputs = self.model( 2025-12-04T09:50:03.6774322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6774411Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6774644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6774733Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6774983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6775058Z return func(*args, **kwargs) 2025-12-04T09:50:03.6775317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6775421Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6775688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6775772Z return func(*args, **kwargs) 2025-12-04T09:50:03.6776025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6776151Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6776155Z 2025-12-04T09:50:03.6776262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6776467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6776560Z res = mod(**inputs) 2025-12-04T09:50:03.6776810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6776886Z outputs = self.model( 2025-12-04T09:50:03.6777138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6777216Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6777450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6777532Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6777779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6777860Z return func(*args, **kwargs) 2025-12-04T09:50:03.6778116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6778228Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6778478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6778550Z return func(*args, **kwargs) 2025-12-04T09:50:03.6778815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6778904Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6778908Z 2025-12-04T09:50:03.6779024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6779232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6779299Z res = mod(**inputs) 2025-12-04T09:50:03.6779563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6779634Z outputs = self.model( 2025-12-04T09:50:03.6779895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6779981Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6780223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6780330Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6780604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6780680Z return func(*args, **kwargs) 2025-12-04T09:50:03.6780956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6781064Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6781324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6781406Z return func(*args, **kwargs) 2025-12-04T09:50:03.6781672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6781802Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6781823Z 2025-12-04T09:50:03.6781936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6782154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6782231Z res = mod(**inputs) 2025-12-04T09:50:03.6782501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6782581Z outputs = self.model( 2025-12-04T09:50:03.6782844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6782944Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6783191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6783276Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6783541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6783625Z return func(*args, **kwargs) 2025-12-04T09:50:03.6783893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6784007Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6784271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6784345Z return func(*args, **kwargs) 2025-12-04T09:50:03.6784620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6784772Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6784776Z 2025-12-04T09:50:03.6784895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6785111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6785183Z res = mod(**inputs) 2025-12-04T09:50:03.6785461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6785533Z outputs = self.model( 2025-12-04T09:50:03.6785799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6785887Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6786131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6786222Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6786497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6786572Z return func(*args, **kwargs) 2025-12-04T09:50:03.6786852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6786990Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6787262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6787343Z return func(*args, **kwargs) 2025-12-04T09:50:03.6787615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6787715Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6787720Z 2025-12-04T09:50:03.6787832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6788053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6788128Z res = mod(**inputs) 2025-12-04T09:50:03.6788408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6788492Z outputs = self.model( 2025-12-04T09:50:03.6788761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6788841Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6789082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6789166Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6789432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6789535Z return func(*args, **kwargs) 2025-12-04T09:50:03.6789796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6789907Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6790176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6790254Z return func(*args, **kwargs) 2025-12-04T09:50:03.6790536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6790639Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6790643Z 2025-12-04T09:50:03.6790761Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6790979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6791052Z res = mod(**inputs) 2025-12-04T09:50:03.6791322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6791394Z outputs = self.model( 2025-12-04T09:50:03.6791693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6791782Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6792065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6792153Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6792393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6792462Z return func(*args, **kwargs) 2025-12-04T09:50:03.6792720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6792821Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6793065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6793142Z return func(*args, **kwargs) 2025-12-04T09:50:03.6793392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6793566Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6793571Z 2025-12-04T09:50:03.6793674Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6793877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6793944Z res = mod(**inputs) 2025-12-04T09:50:03.6794192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6794268Z outputs = self.model( 2025-12-04T09:50:03.6794517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6794590Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6794833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6794913Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6795161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6795236Z return func(*args, **kwargs) 2025-12-04T09:50:03.6795484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6795585Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6795832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6795918Z return func(*args, **kwargs) 2025-12-04T09:50:03.6796169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6796251Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6796254Z 2025-12-04T09:50:03.6796361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6796559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6796621Z res = mod(**inputs) 2025-12-04T09:50:03.6796872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6796944Z outputs = self.model( 2025-12-04T09:50:03.6797205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6797292Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6797526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6797615Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6797873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6797948Z return func(*args, **kwargs) 2025-12-04T09:50:03.6798218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6798346Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6798350Z 2025-12-04T09:50:03.6798460Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6798676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6798747Z res = mod(**inputs) 2025-12-04T09:50:03.6799017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6799089Z outputs = self.model( 2025-12-04T09:50:03.6799354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6799465Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6799724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6799816Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6800078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6800154Z return func(*args, **kwargs) 2025-12-04T09:50:03.6800427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6800645Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6800886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6800972Z return self.act(input) 2025-12-04T09:50:03.6800976Z 2025-12-04T09:50:03.6801114Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6801347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6801421Z res = mod(**inputs) 2025-12-04T09:50:03.6801702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6801785Z outputs = self.model( 2025-12-04T09:50:03.6802050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6802151Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6802401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6802481Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6802741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6802809Z return func(*args, **kwargs) 2025-12-04T09:50:03.6803065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6803152Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6803156Z 2025-12-04T09:50:03.6803259Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6803469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6803536Z res = mod(**inputs) 2025-12-04T09:50:03.6803803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6803881Z outputs = self.model( 2025-12-04T09:50:03.6804151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6804230Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6804481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6804568Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6804841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6804915Z return func(*args, **kwargs) 2025-12-04T09:50:03.6805183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:50:03.6805276Z hidden_states = residual + hidden_states 2025-12-04T09:50:03.6805282Z 2025-12-04T09:50:03.6805401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6805611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6805675Z res = mod(**inputs) 2025-12-04T09:50:03.6805933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6806027Z outputs = self.model( 2025-12-04T09:50:03.6806292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6806368Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6806604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6806683Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6806939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6807013Z return func(*args, **kwargs) 2025-12-04T09:50:03.6807265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6807388Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6807633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6807708Z return func(*args, **kwargs) 2025-12-04T09:50:03.6807977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6808095Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6808099Z 2025-12-04T09:50:03.6808215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6808424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6808512Z res = mod(**inputs) 2025-12-04T09:50:03.6808779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6808852Z outputs = self.model( 2025-12-04T09:50:03.6809123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6809204Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6809440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6809529Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6809785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6809854Z return func(*args, **kwargs) 2025-12-04T09:50:03.6810115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6810221Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6810484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6810559Z return func(*args, **kwargs) 2025-12-04T09:50:03.6810822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6810910Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6810913Z 2025-12-04T09:50:03.6811020Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6811230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6811303Z res = mod(**inputs) 2025-12-04T09:50:03.6811565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6811644Z outputs = self.model( 2025-12-04T09:50:03.6811906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6811985Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6812232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6812355Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6812621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6812694Z return func(*args, **kwargs) 2025-12-04T09:50:03.6812956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6813068Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6813329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6813404Z return func(*args, **kwargs) 2025-12-04T09:50:03.6813674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6813812Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6813818Z 2025-12-04T09:50:03.6813937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6814152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6814219Z res = mod(**inputs) 2025-12-04T09:50:03.6814486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6814559Z outputs = self.model( 2025-12-04T09:50:03.6814830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6814940Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6815175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6815267Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6815531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6815609Z return func(*args, **kwargs) 2025-12-04T09:50:03.6815883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6815988Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6816251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6816327Z return func(*args, **kwargs) 2025-12-04T09:50:03.6816592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6816745Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6816749Z 2025-12-04T09:50:03.6816859Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6817078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6817152Z res = mod(**inputs) 2025-12-04T09:50:03.6817415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6817495Z outputs = self.model( 2025-12-04T09:50:03.6817758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6817834Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6818072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6818156Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6818416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6818490Z return func(*args, **kwargs) 2025-12-04T09:50:03.6818771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6818900Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6819160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6819234Z return func(*args, **kwargs) 2025-12-04T09:50:03.6819513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6819607Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6819611Z 2025-12-04T09:50:03.6819729Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6819945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6820011Z res = mod(**inputs) 2025-12-04T09:50:03.6820286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6820360Z outputs = self.model( 2025-12-04T09:50:03.6820614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6820689Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6820911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6820994Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6821257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6821323Z return func(*args, **kwargs) 2025-12-04T09:50:03.6821574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6821669Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6821919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6821991Z return func(*args, **kwargs) 2025-12-04T09:50:03.6822237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6822345Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6822348Z 2025-12-04T09:50:03.6822451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6822652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6822727Z res = mod(**inputs) 2025-12-04T09:50:03.6822990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6823068Z outputs = self.model( 2025-12-04T09:50:03.6823328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6823408Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6823652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6823737Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6823999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6824073Z return func(*args, **kwargs) 2025-12-04T09:50:03.6824337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6824448Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6824714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6824784Z return func(*args, **kwargs) 2025-12-04T09:50:03.6825075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6825207Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6825211Z 2025-12-04T09:50:03.6825318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6825519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6825584Z res = mod(**inputs) 2025-12-04T09:50:03.6825839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6825910Z outputs = self.model( 2025-12-04T09:50:03.6826165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6826238Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6826477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6826570Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6826810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6826881Z return func(*args, **kwargs) 2025-12-04T09:50:03.6827136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6827235Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6827509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6827578Z return func(*args, **kwargs) 2025-12-04T09:50:03.6827829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6827918Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6827923Z 2025-12-04T09:50:03.6828025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6828222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6828289Z res = mod(**inputs) 2025-12-04T09:50:03.6828547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6828626Z outputs = self.model( 2025-12-04T09:50:03.6828888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6828968Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6829212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6829296Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6829575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6829653Z return func(*args, **kwargs) 2025-12-04T09:50:03.6829916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6830048Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6830052Z 2025-12-04T09:50:03.6830161Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6830382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6830461Z res = mod(**inputs) 2025-12-04T09:50:03.6830724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6830803Z outputs = self.model( 2025-12-04T09:50:03.6831071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6831185Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6831437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6831521Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6831796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6831878Z return func(*args, **kwargs) 2025-12-04T09:50:03.6832193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6832323Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6832546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6832632Z return self.act(input) 2025-12-04T09:50:03.6832636Z 2025-12-04T09:50:03.6832752Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6832955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6833026Z res = mod(**inputs) 2025-12-04T09:50:03.6833279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6833347Z outputs = self.model( 2025-12-04T09:50:03.6833618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6833716Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6833959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6834051Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6834312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6834390Z return func(*args, **kwargs) 2025-12-04T09:50:03.6834638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6834721Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6834725Z 2025-12-04T09:50:03.6834834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6835034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6835109Z res = mod(**inputs) 2025-12-04T09:50:03.6835356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6835424Z outputs = self.model( 2025-12-04T09:50:03.6835677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6835752Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6835978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6836068Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6836316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6836395Z return func(*args, **kwargs) 2025-12-04T09:50:03.6836656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6836763Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6837027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6837098Z return func(*args, **kwargs) 2025-12-04T09:50:03.6837366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6837523Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6837528Z 2025-12-04T09:50:03.6837636Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6837847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6837916Z res = mod(**inputs) 2025-12-04T09:50:03.6838177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6838260Z outputs = self.model( 2025-12-04T09:50:03.6838524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6838611Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6838872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6838958Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6839228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6839303Z return func(*args, **kwargs) 2025-12-04T09:50:03.6839573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6839680Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6839946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6840046Z return func(*args, **kwargs) 2025-12-04T09:50:03.6840316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6840402Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6840406Z 2025-12-04T09:50:03.6840611Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6840837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6840914Z res = mod(**inputs) 2025-12-04T09:50:03.6841190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6841266Z outputs = self.model( 2025-12-04T09:50:03.6841547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6841631Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6841879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6841972Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6842232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6842317Z return func(*args, **kwargs) 2025-12-04T09:50:03.6842582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6842689Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6842955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6843030Z return func(*args, **kwargs) 2025-12-04T09:50:03.6843301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6843424Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6843428Z 2025-12-04T09:50:03.6843538Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6843757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6843828Z res = mod(**inputs) 2025-12-04T09:50:03.6844132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6844212Z outputs = self.model( 2025-12-04T09:50:03.6844473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6844558Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6844795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6844882Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6845148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6845221Z return func(*args, **kwargs) 2025-12-04T09:50:03.6845527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6845638Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6845898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6845979Z return func(*args, **kwargs) 2025-12-04T09:50:03.6846241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6846385Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6846407Z 2025-12-04T09:50:03.6846526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6846740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6846816Z res = mod(**inputs) 2025-12-04T09:50:03.6847395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6847481Z outputs = self.model( 2025-12-04T09:50:03.6847764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6847845Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6848084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6848176Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6848439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6848524Z return func(*args, **kwargs) 2025-12-04T09:50:03.6848790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6848894Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6849167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6849246Z return func(*args, **kwargs) 2025-12-04T09:50:03.6849525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6849621Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6849625Z 2025-12-04T09:50:03.6849734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6849955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6850028Z res = mod(**inputs) 2025-12-04T09:50:03.6850296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6850380Z outputs = self.model( 2025-12-04T09:50:03.6850646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6851054Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6851318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6851401Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6851661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6851732Z return func(*args, **kwargs) 2025-12-04T09:50:03.6851987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6852091Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6852335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6852415Z return func(*args, **kwargs) 2025-12-04T09:50:03.6852697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6852800Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6852804Z 2025-12-04T09:50:03.6852918Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6853117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6853189Z res = mod(**inputs) 2025-12-04T09:50:03.6853441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6853535Z outputs = self.model( 2025-12-04T09:50:03.6853795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6853873Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6854100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6854188Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6854429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6854506Z return func(*args, **kwargs) 2025-12-04T09:50:03.6854750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6854849Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6855091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6855158Z return func(*args, **kwargs) 2025-12-04T09:50:03.6855408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6855536Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6855541Z 2025-12-04T09:50:03.6855644Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6855850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6855917Z res = mod(**inputs) 2025-12-04T09:50:03.6856163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6856241Z outputs = self.model( 2025-12-04T09:50:03.6856487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6856568Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6856788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6856865Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6857114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6857202Z return func(*args, **kwargs) 2025-12-04T09:50:03.6857476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6857579Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6857826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6857901Z return func(*args, **kwargs) 2025-12-04T09:50:03.6858153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6858238Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6858241Z 2025-12-04T09:50:03.6858354Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6858574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6858650Z res = mod(**inputs) 2025-12-04T09:50:03.6858901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6858972Z outputs = self.model( 2025-12-04T09:50:03.6859227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6859301Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6859526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6859628Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6859868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6859943Z return func(*args, **kwargs) 2025-12-04T09:50:03.6860190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6860312Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6860316Z 2025-12-04T09:50:03.6860429Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6860629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6860703Z res = mod(**inputs) 2025-12-04T09:50:03.6860952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6861024Z outputs = self.model( 2025-12-04T09:50:03.6861280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6861357Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6861582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6861672Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6861916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6861992Z return func(*args, **kwargs) 2025-12-04T09:50:03.6862241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6862360Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6862584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6862653Z return self.act(input) 2025-12-04T09:50:03.6862657Z 2025-12-04T09:50:03.6862765Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6862961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6863030Z res = mod(**inputs) 2025-12-04T09:50:03.6863299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6863393Z outputs = self.model( 2025-12-04T09:50:03.6863652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6863736Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6863963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6864049Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6864300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6864371Z return func(*args, **kwargs) 2025-12-04T09:50:03.6864631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6864730Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6864735Z 2025-12-04T09:50:03.6864841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6865046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6865113Z res = mod(**inputs) 2025-12-04T09:50:03.6865366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6865437Z outputs = self.model( 2025-12-04T09:50:03.6865683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6865794Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6866012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6866097Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6866338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6866409Z return func(*args, **kwargs) 2025-12-04T09:50:03.6866663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:50:03.6866747Z hidden_states = residual + hidden_states 2025-12-04T09:50:03.6866750Z 2025-12-04T09:50:03.6866855Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6867063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6867130Z res = mod(**inputs) 2025-12-04T09:50:03.6867387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6867456Z outputs = self.model( 2025-12-04T09:50:03.6867704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6867789Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6868017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6868095Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6868356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6868425Z return func(*args, **kwargs) 2025-12-04T09:50:03.6868673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6868772Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6869035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6869114Z return func(*args, **kwargs) 2025-12-04T09:50:03.6869363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6869518Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6869522Z 2025-12-04T09:50:03.6869629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6869849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6869925Z res = mod(**inputs) 2025-12-04T09:50:03.6870198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6870272Z outputs = self.model( 2025-12-04T09:50:03.6870552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6870632Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6870898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6870988Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6871246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6871328Z return func(*args, **kwargs) 2025-12-04T09:50:03.6871591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6871689Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6871955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6872023Z return func(*args, **kwargs) 2025-12-04T09:50:03.6872279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:50:03.6872361Z key_states = self.k_proj(current_states) 2025-12-04T09:50:03.6872366Z 2025-12-04T09:50:03.6872469Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6872677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6872741Z res = mod(**inputs) 2025-12-04T09:50:03.6873000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6873069Z outputs = self.model( 2025-12-04T09:50:03.6873318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6873403Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6873625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6873703Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6873957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6874030Z return func(*args, **kwargs) 2025-12-04T09:50:03.6874284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6874381Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6874637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6874711Z return func(*args, **kwargs) 2025-12-04T09:50:03.6874956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:50:03.6875074Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:50:03.6875077Z 2025-12-04T09:50:03.6875178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6875373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6875467Z res = mod(**inputs) 2025-12-04T09:50:03.6875728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6875798Z outputs = self.model( 2025-12-04T09:50:03.6876046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6876120Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6876345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6876425Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6876664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6876742Z return func(*args, **kwargs) 2025-12-04T09:50:03.6877000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6877101Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6877344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6877413Z return func(*args, **kwargs) 2025-12-04T09:50:03.6877671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:50:03.6877804Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:50:03.6877825Z 2025-12-04T09:50:03.6877931Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6878138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6878201Z res = mod(**inputs) 2025-12-04T09:50:03.6878458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6878527Z outputs = self.model( 2025-12-04T09:50:03.6878778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6878863Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6879101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6879184Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6879449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6879524Z return func(*args, **kwargs) 2025-12-04T09:50:03.6879795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6879900Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6880160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6880244Z return func(*args, **kwargs) 2025-12-04T09:50:03.6880579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:50:03.6880684Z value_states = self.v_proj(current_states) 2025-12-04T09:50:03.6880688Z 2025-12-04T09:50:03.6880800Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6881011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6881093Z res = mod(**inputs) 2025-12-04T09:50:03.6881360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6881435Z outputs = self.model( 2025-12-04T09:50:03.6881710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6881827Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6882084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6882163Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6882407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6882486Z return func(*args, **kwargs) 2025-12-04T09:50:03.6882741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6882849Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6883121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6883209Z return func(*args, **kwargs) 2025-12-04T09:50:03.6883483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:50:03.6883587Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:50:03.6883590Z 2025-12-04T09:50:03.6883696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6883915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6883985Z res = mod(**inputs) 2025-12-04T09:50:03.6884253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6884347Z outputs = self.model( 2025-12-04T09:50:03.6884622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6884710Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6884956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6885042Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6885314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6885387Z return func(*args, **kwargs) 2025-12-04T09:50:03.6885668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6885772Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6886037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6886118Z return func(*args, **kwargs) 2025-12-04T09:50:03.6886388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:50:03.6886532Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:50:03.6886537Z 2025-12-04T09:50:03.6886648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6886862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6886937Z res = mod(**inputs) 2025-12-04T09:50:03.6887203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6887277Z outputs = self.model( 2025-12-04T09:50:03.6887553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6887634Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6887883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6887966Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6888232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6888359Z return func(*args, **kwargs) 2025-12-04T09:50:03.6888625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:50:03.6888728Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:50:03.6888993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6889066Z return func(*args, **kwargs) 2025-12-04T09:50:03.6889341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:50:03.6889428Z attn_output = self.out_proj(attn_output) 2025-12-04T09:50:03.6889432Z 2025-12-04T09:50:03.6889541Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6889789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6889861Z res = mod(**inputs) 2025-12-04T09:50:03.6890145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6890219Z outputs = self.model( 2025-12-04T09:50:03.6890491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6890577Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6890822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6890925Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6891198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6891271Z return func(*args, **kwargs) 2025-12-04T09:50:03.6891554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6891684Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6891687Z 2025-12-04T09:50:03.6891815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6892039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6892115Z res = mod(**inputs) 2025-12-04T09:50:03.6892412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6892482Z outputs = self.model( 2025-12-04T09:50:03.6892743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6892817Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6893045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6893139Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6893407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6893490Z return func(*args, **kwargs) 2025-12-04T09:50:03.6893764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:50:03.6893890Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:50:03.6894132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:03.6894208Z return self.act(input) 2025-12-04T09:50:03.6894212Z 2025-12-04T09:50:03.6894338Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6894539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6894605Z res = mod(**inputs) 2025-12-04T09:50:03.6894893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:50:03.6894963Z outputs = self.model( 2025-12-04T09:50:03.6895223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:50:03.6895304Z layer_outputs = decoder_layer( 2025-12-04T09:50:03.6895529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:50:03.6895618Z return super().__call__(*args, **kwargs) 2025-12-04T09:50:03.6895864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:50:03.6895935Z return func(*args, **kwargs) 2025-12-04T09:50:03.6896207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:50:03.6896295Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:50:03.6896300Z 2025-12-04T09:50:03.6896411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6896612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6896677Z res = mod(**inputs) 2025-12-04T09:50:03.6896933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 685, in forward 2025-12-04T09:50:03.6897012Z logits = self.lm_head(outputs[0]) 2025-12-04T09:50:03.6897033Z 2025-12-04T09:50:03.6897137Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:03.6897347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:03.6897412Z res = mod(**inputs) 2025-12-04T09:50:03.6897670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 689, in forward 2025-12-04T09:50:03.6897749Z loss = self.loss_function( 2025-12-04T09:50:03.6897996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:50:03.6898181Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:50:03.6898433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:50:03.6898629Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:50:03.6898639Z 2025-12-04T09:50:16.0262799Z Compilation time (from dynamo_timed): 26.031828703 2025-12-04T09:50:16.0356299Z pass 2025-12-04T09:50:16.0356860Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:50:16.0357951Z TIMING: _recursive_pre_grad_passes:0.01278 _recursive_joint_graph_passes:0.76408 _recursive_post_grad_passes:0.24008 async_compile.wait:0.80809 code_gen:12.3845 inductor_compile:15.71761 backend_compile:21.16578 gc:0.00041 entire_frame_compile:26.03183 total_wall_time:26.03183 2025-12-04T09:50:16.0361073Z STATS: call_* op count: 921 | FakeTensorMode.__torch_dispatch__:17015 | FakeTensor.__torch_dispatch__:9977 | ProxyTorchDispatchMode.__torch_dispatch__:4882 2025-12-04T09:50:16.0361651Z Dynamo produced 1 graphs covering 921 ops with 0 graph breaks (0 unique) 2025-12-04T09:50:18.9021849Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:50:18.9023096Z import pynvml # type: ignore[import] 2025-12-04T09:50:22.3482586Z 2025-12-04T09:50:25.1025081Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:50:25.1025567Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:50:25.1043200Z cpu eval XLNetLMHeadModel 2025-12-04T09:50:27.8541957Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:50:28.8229127Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:50:29.7807448Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:50:51.7886760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7889260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7889793Z res = mod(**inputs) 2025-12-04T09:50:51.7890308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7890876Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7891784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1307, in forward 2025-12-04T09:50:51.7892270Z word_emb_k = self.word_embedding(input_ids) 2025-12-04T09:50:51.7892554Z 2025-12-04T09:50:51.7893357Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7893846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7894237Z res = mod(**inputs) 2025-12-04T09:50:51.7894703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7895471Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7896064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-12-04T09:50:51.7896591Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-12-04T09:50:51.7897160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-12-04T09:50:51.7897726Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-12-04T09:50:51.7898278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-12-04T09:50:51.7898855Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-12-04T09:50:51.7899094Z 2025-12-04T09:50:51.7899233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7899653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7900022Z res = mod(**inputs) 2025-12-04T09:50:51.7900450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7900921Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7901381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-12-04T09:50:51.7901885Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-12-04T09:50:51.7902447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-12-04T09:50:51.7903010Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-12-04T09:50:51.7903556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-12-04T09:50:51.7904140Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-12-04T09:50:51.7904466Z 2025-12-04T09:50:51.7904594Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7905041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7905548Z res = mod(**inputs) 2025-12-04T09:50:51.7906119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7906598Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7907056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7907511Z outputs = layer_module( 2025-12-04T09:50:51.7907933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7908384Z outputs = self.rel_attn( 2025-12-04T09:50:51.7908803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7909278Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7909800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7910307Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7910508Z 2025-12-04T09:50:51.7910629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7911042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7911408Z res = mod(**inputs) 2025-12-04T09:50:51.7911824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7912334Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7912797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7913245Z outputs = layer_module( 2025-12-04T09:50:51.7913659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7914106Z outputs = self.rel_attn( 2025-12-04T09:50:51.7914520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7914967Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7915439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7915948Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7916143Z 2025-12-04T09:50:51.7916267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7916661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7917031Z res = mod(**inputs) 2025-12-04T09:50:51.7917450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7917905Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7918343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7918776Z outputs = layer_module( 2025-12-04T09:50:51.7919253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7919686Z outputs = self.rel_attn( 2025-12-04T09:50:51.7920103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7920904Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7921404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7921920Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7922197Z 2025-12-04T09:50:51.7922350Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7922764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7923134Z res = mod(**inputs) 2025-12-04T09:50:51.7923546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7924000Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7924449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7924893Z outputs = layer_module( 2025-12-04T09:50:51.7925314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7925784Z outputs = self.rel_attn( 2025-12-04T09:50:51.7926246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7926692Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7927159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7927650Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7927832Z 2025-12-04T09:50:51.7927955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7928429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7928786Z res = mod(**inputs) 2025-12-04T09:50:51.7929187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7929626Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7930081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7930516Z outputs = layer_module( 2025-12-04T09:50:51.7930934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7931364Z outputs = self.rel_attn( 2025-12-04T09:50:51.7931779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7932222Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7932686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7933189Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7933378Z 2025-12-04T09:50:51.7933497Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7933897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7934248Z res = mod(**inputs) 2025-12-04T09:50:51.7934654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7935105Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7935543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7935985Z outputs = layer_module( 2025-12-04T09:50:51.7936400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7936830Z outputs = self.rel_attn( 2025-12-04T09:50:51.7937233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7937688Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7938219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7938722Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7938907Z 2025-12-04T09:50:51.7939024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7939433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7939800Z res = mod(**inputs) 2025-12-04T09:50:51.7940199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7940658Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7941101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7941598Z outputs = layer_module( 2025-12-04T09:50:51.7942007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7942444Z outputs = self.rel_attn( 2025-12-04T09:50:51.7942869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7943315Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7943772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7944296Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7944483Z 2025-12-04T09:50:51.7944609Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7945011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7945371Z res = mod(**inputs) 2025-12-04T09:50:51.7945784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7946252Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7946699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7947471Z outputs = layer_module( 2025-12-04T09:50:51.7947888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7948324Z outputs = self.rel_attn( 2025-12-04T09:50:51.7948736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7949193Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7949664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7950157Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7950349Z 2025-12-04T09:50:51.7950468Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7950864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7951215Z res = mod(**inputs) 2025-12-04T09:50:51.7951611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7952060Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7952518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7952949Z outputs = layer_module( 2025-12-04T09:50:51.7953361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7953789Z outputs = self.rel_attn( 2025-12-04T09:50:51.7954337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7954792Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7955269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7955772Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7955955Z 2025-12-04T09:50:51.7956081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7956477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7956836Z res = mod(**inputs) 2025-12-04T09:50:51.7957241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7957711Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7958159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7958586Z outputs = layer_module( 2025-12-04T09:50:51.7959013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7959445Z outputs = self.rel_attn( 2025-12-04T09:50:51.7959880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7960361Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7960933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7961438Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7961637Z 2025-12-04T09:50:51.7961761Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7962171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7962526Z res = mod(**inputs) 2025-12-04T09:50:51.7962958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7963421Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7963880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7964312Z outputs = layer_module( 2025-12-04T09:50:51.7964748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7965191Z outputs = self.rel_attn( 2025-12-04T09:50:51.7965620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7966079Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7966552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7967047Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7967233Z 2025-12-04T09:50:51.7967349Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7967752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7968126Z res = mod(**inputs) 2025-12-04T09:50:51.7968537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7968974Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7969415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7969845Z outputs = layer_module( 2025-12-04T09:50:51.7970309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7970739Z outputs = self.rel_attn( 2025-12-04T09:50:51.7971150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7971586Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7972027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7972509Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7972688Z 2025-12-04T09:50:51.7972816Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7973213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7973577Z res = mod(**inputs) 2025-12-04T09:50:51.7973986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7974433Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7974871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7975303Z outputs = layer_module( 2025-12-04T09:50:51.7975714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7976155Z outputs = self.rel_attn( 2025-12-04T09:50:51.7976548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7976992Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7977518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7978004Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7978183Z 2025-12-04T09:50:51.7978295Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7978681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7979028Z res = mod(**inputs) 2025-12-04T09:50:51.7979409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7979844Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7980269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7980702Z outputs = layer_module( 2025-12-04T09:50:51.7981130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7981565Z outputs = self.rel_attn( 2025-12-04T09:50:51.7981985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7982418Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7982896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7983395Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7983575Z 2025-12-04T09:50:51.7983699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7984092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7984450Z res = mod(**inputs) 2025-12-04T09:50:51.7984861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7985308Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7985787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7986222Z outputs = layer_module( 2025-12-04T09:50:51.7986632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7987069Z outputs = self.rel_attn( 2025-12-04T09:50:51.7987503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7987961Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7988434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7988926Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7989152Z 2025-12-04T09:50:51.7989272Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7989676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7990035Z res = mod(**inputs) 2025-12-04T09:50:51.7990436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7990884Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7991330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7991772Z outputs = layer_module( 2025-12-04T09:50:51.7992189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7992627Z outputs = self.rel_attn( 2025-12-04T09:50:51.7993051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7993497Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7993977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.7994481Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.7994671Z 2025-12-04T09:50:51.7994798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.7995195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.7995562Z res = mod(**inputs) 2025-12-04T09:50:51.7995973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.7996418Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.7996870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.7997305Z outputs = layer_module( 2025-12-04T09:50:51.7997724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.7998155Z outputs = self.rel_attn( 2025-12-04T09:50:51.7998576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.7999065Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.7999532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8000036Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8000227Z 2025-12-04T09:50:51.8000345Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8000814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8001244Z res = mod(**inputs) 2025-12-04T09:50:51.8001663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8002118Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8002562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8003004Z outputs = layer_module( 2025-12-04T09:50:51.8003426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8003862Z outputs = self.rel_attn( 2025-12-04T09:50:51.8004272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8004734Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8005234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8005734Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8005917Z 2025-12-04T09:50:51.8006033Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8006430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8006789Z res = mod(**inputs) 2025-12-04T09:50:51.8007192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8007651Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8008117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8008564Z outputs = layer_module( 2025-12-04T09:50:51.8008974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8009429Z outputs = self.rel_attn( 2025-12-04T09:50:51.8009858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8010325Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8010786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8011279Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8011462Z 2025-12-04T09:50:51.8011582Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8011974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8012334Z res = mod(**inputs) 2025-12-04T09:50:51.8012739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8013186Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8013639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8014080Z outputs = layer_module( 2025-12-04T09:50:51.8014502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8014942Z outputs = self.rel_attn( 2025-12-04T09:50:51.8015358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8015815Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8016286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8016774Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8016990Z 2025-12-04T09:50:51.8017104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8017536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8017898Z res = mod(**inputs) 2025-12-04T09:50:51.8018296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8018745Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8019192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8019626Z outputs = layer_module( 2025-12-04T09:50:51.8020054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8020492Z outputs = self.rel_attn( 2025-12-04T09:50:51.8020930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8021368Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8021831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8022314Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8022496Z 2025-12-04T09:50:51.8022617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8023005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8023382Z res = mod(**inputs) 2025-12-04T09:50:51.8023788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8024233Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8024695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8025122Z outputs = layer_module( 2025-12-04T09:50:51.8025527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8025956Z outputs = self.rel_attn( 2025-12-04T09:50:51.8026367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8026824Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8027296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8027785Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8027972Z 2025-12-04T09:50:51.8028086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8028485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8028836Z res = mod(**inputs) 2025-12-04T09:50:51.8029242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8029684Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8030123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8030537Z outputs = layer_module( 2025-12-04T09:50:51.8030943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8031370Z outputs = self.rel_attn( 2025-12-04T09:50:51.8031771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8032213Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8032703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8033218Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8033401Z 2025-12-04T09:50:51.8033517Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8033919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8034276Z res = mod(**inputs) 2025-12-04T09:50:51.8034683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8035120Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8035564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8035993Z outputs = layer_module( 2025-12-04T09:50:51.8036418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8036853Z outputs = self.rel_attn( 2025-12-04T09:50:51.8037267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8037714Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8038173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8038665Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8038869Z 2025-12-04T09:50:51.8038993Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8039390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8039740Z res = mod(**inputs) 2025-12-04T09:50:51.8040144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8040678Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8041123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8041559Z outputs = layer_module( 2025-12-04T09:50:51.8041967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8042393Z outputs = self.rel_attn( 2025-12-04T09:50:51.8042797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8043253Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8043422Z 2025-12-04T09:50:51.8043547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8043937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8044297Z res = mod(**inputs) 2025-12-04T09:50:51.8044693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8045136Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8045591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8046030Z outputs = layer_module( 2025-12-04T09:50:51.8046437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8046862Z outputs = self.rel_attn( 2025-12-04T09:50:51.8047463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8047939Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8048118Z 2025-12-04T09:50:51.8048308Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8048746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8049102Z res = mod(**inputs) 2025-12-04T09:50:51.8049526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8049974Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8050409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8050846Z outputs = layer_module( 2025-12-04T09:50:51.8051263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8051684Z outputs = self.rel_attn( 2025-12-04T09:50:51.8052126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8052566Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8053030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8053537Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8053752Z 2025-12-04T09:50:51.8053868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8054260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8054654Z res = mod(**inputs) 2025-12-04T09:50:51.8055036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8055478Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8055936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-12-04T09:50:51.8056435Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-12-04T09:50:51.8056965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-12-04T09:50:51.8057502Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-12-04T09:50:51.8058016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-12-04T09:50:51.8058554Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-12-04T09:50:51.8058786Z 2025-12-04T09:50:51.8058904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8059305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8059664Z res = mod(**inputs) 2025-12-04T09:50:51.8060065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8060507Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8060953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8061379Z outputs = layer_module( 2025-12-04T09:50:51.8061781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8062211Z outputs = self.rel_attn( 2025-12-04T09:50:51.8062620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8063121Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8063339Z 2025-12-04T09:50:51.8063456Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8063849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8064250Z res = mod(**inputs) 2025-12-04T09:50:51.8064646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8065093Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8065528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8065942Z outputs = layer_module( 2025-12-04T09:50:51.8066339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8066760Z outputs = self.rel_attn( 2025-12-04T09:50:51.8067159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8067604Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8068040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8068545Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8068745Z 2025-12-04T09:50:51.8068866Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8069244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8069593Z res = mod(**inputs) 2025-12-04T09:50:51.8070003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8070427Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8070857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8071271Z outputs = layer_module( 2025-12-04T09:50:51.8071673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8072079Z outputs = self.rel_attn( 2025-12-04T09:50:51.8072479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8072934Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8073128Z 2025-12-04T09:50:51.8073250Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8073637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8074004Z res = mod(**inputs) 2025-12-04T09:50:51.8074395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8074820Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8075250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8075679Z outputs = layer_module( 2025-12-04T09:50:51.8076086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8076503Z outputs = self.rel_attn( 2025-12-04T09:50:51.8076915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8077342Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8077779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8078296Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8078499Z 2025-12-04T09:50:51.8078612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8079021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8079417Z res = mod(**inputs) 2025-12-04T09:50:51.8079815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8080256Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8080780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8081219Z outputs = layer_module( 2025-12-04T09:50:51.8081635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8082065Z outputs = self.rel_attn( 2025-12-04T09:50:51.8082471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8082961Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8083436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8083934Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8084120Z 2025-12-04T09:50:51.8084238Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8084637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8084993Z res = mod(**inputs) 2025-12-04T09:50:51.8085395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8085860Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8086302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8086729Z outputs = layer_module( 2025-12-04T09:50:51.8087155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8087583Z outputs = self.rel_attn( 2025-12-04T09:50:51.8087994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8088453Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8088910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8089381Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8089549Z 2025-12-04T09:50:51.8089665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8090043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8090387Z res = mod(**inputs) 2025-12-04T09:50:51.8090776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8091210Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8091631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8092044Z outputs = layer_module( 2025-12-04T09:50:51.8092441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8093003Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8093571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8094002Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8094426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8094866Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8095309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8095728Z output = self.layer_1(output) 2025-12-04T09:50:51.8095865Z 2025-12-04T09:50:51.8095984Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8096363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8096712Z res = mod(**inputs) 2025-12-04T09:50:51.8097101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8097529Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8097977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8098402Z outputs = layer_module( 2025-12-04T09:50:51.8098803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8099362Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8099940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8100378Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8100827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8101245Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8101657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8102112Z output = self.activation_function(output) 2025-12-04T09:50:51.8102504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8102887Z return self.act(input) 2025-12-04T09:50:51.8103017Z 2025-12-04T09:50:51.8103755Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8104145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8104488Z res = mod(**inputs) 2025-12-04T09:50:51.8104908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8105345Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8105793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8106215Z outputs = layer_module( 2025-12-04T09:50:51.8106616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8107179Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8107750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8108189Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8108616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8109042Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8109446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8109885Z output = self.layer_2(output) 2025-12-04T09:50:51.8110030Z 2025-12-04T09:50:51.8110142Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8110532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8110930Z res = mod(**inputs) 2025-12-04T09:50:51.8111326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8111772Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8112192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8112603Z outputs = layer_module( 2025-12-04T09:50:51.8113000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8113416Z outputs = self.rel_attn( 2025-12-04T09:50:51.8113808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8114274Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8114442Z 2025-12-04T09:50:51.8114564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8114949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8115288Z res = mod(**inputs) 2025-12-04T09:50:51.8115675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8116104Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8116527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8116965Z outputs = layer_module( 2025-12-04T09:50:51.8117364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8117778Z outputs = self.rel_attn( 2025-12-04T09:50:51.8118173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8118639Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8118812Z 2025-12-04T09:50:51.8118935Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8119324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8119682Z res = mod(**inputs) 2025-12-04T09:50:51.8120066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8120499Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8121022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8121459Z outputs = layer_module( 2025-12-04T09:50:51.8121874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8122318Z outputs = self.rel_attn( 2025-12-04T09:50:51.8122714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8123135Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8123588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8124099Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8124318Z 2025-12-04T09:50:51.8124438Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8124840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8125220Z res = mod(**inputs) 2025-12-04T09:50:51.8125638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8126126Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8126567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8126995Z outputs = layer_module( 2025-12-04T09:50:51.8127407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8127838Z outputs = self.rel_attn( 2025-12-04T09:50:51.8128248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8128742Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8128961Z 2025-12-04T09:50:51.8129078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8129492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8129858Z res = mod(**inputs) 2025-12-04T09:50:51.8130272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8130711Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8131166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8131593Z outputs = layer_module( 2025-12-04T09:50:51.8132003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8132452Z outputs = self.rel_attn( 2025-12-04T09:50:51.8132860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8133284Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8133729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8134243Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8134449Z 2025-12-04T09:50:51.8134572Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8134960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8135314Z res = mod(**inputs) 2025-12-04T09:50:51.8135718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8136165Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8136593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8137009Z outputs = layer_module( 2025-12-04T09:50:51.8137412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8137830Z outputs = self.rel_attn( 2025-12-04T09:50:51.8138229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8138678Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8138846Z 2025-12-04T09:50:51.8138959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8139347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8139695Z res = mod(**inputs) 2025-12-04T09:50:51.8140086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8140512Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8140941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8141381Z outputs = layer_module( 2025-12-04T09:50:51.8141811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8142219Z outputs = self.rel_attn( 2025-12-04T09:50:51.8142622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8143061Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8143496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8143989Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8144186Z 2025-12-04T09:50:51.8144298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8144714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8145054Z res = mod(**inputs) 2025-12-04T09:50:51.8145443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8145889Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8146344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8146769Z outputs = layer_module( 2025-12-04T09:50:51.8147382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8147872Z outputs = self.rel_attn( 2025-12-04T09:50:51.8148270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8148723Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8149199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8149702Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8149889Z 2025-12-04T09:50:51.8150006Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8150408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8150779Z res = mod(**inputs) 2025-12-04T09:50:51.8151182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8151633Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8152079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8152518Z outputs = layer_module( 2025-12-04T09:50:51.8152922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8153354Z outputs = self.rel_attn( 2025-12-04T09:50:51.8153769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8154219Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8154684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8155184Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8155369Z 2025-12-04T09:50:51.8155492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8155884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8156249Z res = mod(**inputs) 2025-12-04T09:50:51.8159480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8159974Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8160428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8160937Z outputs = layer_module( 2025-12-04T09:50:51.8161354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8161928Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8162522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8162973Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8163444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8163927Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8164351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8164785Z output = self.layer_1(output) 2025-12-04T09:50:51.8164933Z 2025-12-04T09:50:51.8165051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8165456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8165807Z res = mod(**inputs) 2025-12-04T09:50:51.8166214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8166705Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8167159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8167590Z outputs = layer_module( 2025-12-04T09:50:51.8168003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8168592Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8169161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8169587Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8170005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8170428Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8170830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8171276Z output = self.activation_function(output) 2025-12-04T09:50:51.8171670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8172045Z return self.act(input) 2025-12-04T09:50:51.8172165Z 2025-12-04T09:50:51.8172280Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8172667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8173010Z res = mod(**inputs) 2025-12-04T09:50:51.8173399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8173823Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8174259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8174684Z outputs = layer_module( 2025-12-04T09:50:51.8175084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8176339Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8176930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8177375Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8177788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8178218Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8178638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8179073Z output = self.layer_2(output) 2025-12-04T09:50:51.8179208Z 2025-12-04T09:50:51.8179322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8179757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8180123Z res = mod(**inputs) 2025-12-04T09:50:51.8180517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8180961Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8181404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8181841Z outputs = layer_module( 2025-12-04T09:50:51.8182234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8182674Z outputs = self.rel_attn( 2025-12-04T09:50:51.8183081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8183535Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8183717Z 2025-12-04T09:50:51.8183834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8184228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8184584Z res = mod(**inputs) 2025-12-04T09:50:51.8184970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8185414Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8185846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8186267Z outputs = layer_module( 2025-12-04T09:50:51.8186660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8187084Z outputs = self.rel_attn( 2025-12-04T09:50:51.8187494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8187942Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8188123Z 2025-12-04T09:50:51.8188238Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8188625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8188976Z res = mod(**inputs) 2025-12-04T09:50:51.8189363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8189803Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8190235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8190657Z outputs = layer_module( 2025-12-04T09:50:51.8191063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8191544Z outputs = self.rel_attn( 2025-12-04T09:50:51.8191954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8192388Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8192834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8193358Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8193563Z 2025-12-04T09:50:51.8193687Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8194075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8194427Z res = mod(**inputs) 2025-12-04T09:50:51.8194823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8195277Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8195705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8196114Z outputs = layer_module( 2025-12-04T09:50:51.8196504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8196909Z outputs = self.rel_attn( 2025-12-04T09:50:51.8197305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8197810Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8198016Z 2025-12-04T09:50:51.8198139Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8198531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8198886Z res = mod(**inputs) 2025-12-04T09:50:51.8199284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8199718Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8200151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8200664Z outputs = layer_module( 2025-12-04T09:50:51.8201075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8201497Z outputs = self.rel_attn( 2025-12-04T09:50:51.8201911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8202346Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8202791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8203298Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8203506Z 2025-12-04T09:50:51.8203619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8204008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8204353Z res = mod(**inputs) 2025-12-04T09:50:51.8204745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8205191Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8205619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8206036Z outputs = layer_module( 2025-12-04T09:50:51.8206433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8206901Z outputs = self.rel_attn( 2025-12-04T09:50:51.8207297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8207755Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8207931Z 2025-12-04T09:50:51.8208043Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8208430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8208767Z res = mod(**inputs) 2025-12-04T09:50:51.8209154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8209596Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8210010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8210456Z outputs = layer_module( 2025-12-04T09:50:51.8210857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8211270Z outputs = self.rel_attn( 2025-12-04T09:50:51.8211660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8212085Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8212513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8213031Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8213221Z 2025-12-04T09:50:51.8213331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8213719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8214092Z res = mod(**inputs) 2025-12-04T09:50:51.8214474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8214906Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8215337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8215750Z outputs = layer_module( 2025-12-04T09:50:51.8216141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8216562Z outputs = self.rel_attn( 2025-12-04T09:50:51.8216962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8217394Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8217856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8218349Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8218532Z 2025-12-04T09:50:51.8218652Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8219030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8219380Z res = mod(**inputs) 2025-12-04T09:50:51.8219773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8220207Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8220633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8221052Z outputs = layer_module( 2025-12-04T09:50:51.8221454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8221886Z outputs = self.rel_attn( 2025-12-04T09:50:51.8222298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8222732Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8223189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8223677Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8223872Z 2025-12-04T09:50:51.8223987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8224382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8224735Z res = mod(**inputs) 2025-12-04T09:50:51.8225128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8225605Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8226049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8226465Z outputs = layer_module( 2025-12-04T09:50:51.8226893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8227484Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8228078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8228549Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8228989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8229442Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8229881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8230319Z output = self.layer_1(output) 2025-12-04T09:50:51.8230467Z 2025-12-04T09:50:51.8230583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8230990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8231363Z res = mod(**inputs) 2025-12-04T09:50:51.8231784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8232244Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8232696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8233144Z outputs = layer_module( 2025-12-04T09:50:51.8233572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8234165Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8234764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8235211Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8235651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8236100Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8236527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8236980Z output = self.activation_function(output) 2025-12-04T09:50:51.8237396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8237827Z return self.act(input) 2025-12-04T09:50:51.8237971Z 2025-12-04T09:50:51.8238095Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8238489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8238844Z res = mod(**inputs) 2025-12-04T09:50:51.8239246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8239682Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8240122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8240620Z outputs = layer_module( 2025-12-04T09:50:51.8241049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8241652Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8242239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8242684Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8243121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8243555Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8243986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8244481Z output = self.layer_2(output) 2025-12-04T09:50:51.8244615Z 2025-12-04T09:50:51.8244726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8245118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8245476Z res = mod(**inputs) 2025-12-04T09:50:51.8245880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8246304Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8246732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8247356Z outputs = layer_module( 2025-12-04T09:50:51.8247757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8248183Z outputs = self.rel_attn( 2025-12-04T09:50:51.8248584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8249033Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8249201Z 2025-12-04T09:50:51.8249318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8249714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8250066Z res = mod(**inputs) 2025-12-04T09:50:51.8250448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8250882Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8251315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8251734Z outputs = layer_module( 2025-12-04T09:50:51.8252124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8252539Z outputs = self.rel_attn( 2025-12-04T09:50:51.8252935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8253453Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8253647Z 2025-12-04T09:50:51.8253760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8254148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8254495Z res = mod(**inputs) 2025-12-04T09:50:51.8254872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8255307Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8255739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8256154Z outputs = layer_module( 2025-12-04T09:50:51.8256544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8256999Z outputs = self.rel_attn( 2025-12-04T09:50:51.8257405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8257823Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8258250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8258763Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8258969Z 2025-12-04T09:50:51.8259091Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8259525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8259871Z res = mod(**inputs) 2025-12-04T09:50:51.8260258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8260681Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8261100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8261512Z outputs = layer_module( 2025-12-04T09:50:51.8261904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8262308Z outputs = self.rel_attn( 2025-12-04T09:50:51.8262707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8263201Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8263415Z 2025-12-04T09:50:51.8263535Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8263913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8264258Z res = mod(**inputs) 2025-12-04T09:50:51.8264648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8265076Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8265504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8265928Z outputs = layer_module( 2025-12-04T09:50:51.8266332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8266750Z outputs = self.rel_attn( 2025-12-04T09:50:51.8267162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8267595Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8268050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8268582Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8268815Z 2025-12-04T09:50:51.8268929Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8269327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8269705Z res = mod(**inputs) 2025-12-04T09:50:51.8270098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8270544Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8270985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8271413Z outputs = layer_module( 2025-12-04T09:50:51.8271822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8272279Z outputs = self.rel_attn( 2025-12-04T09:50:51.8272693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8273147Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8273327Z 2025-12-04T09:50:51.8273442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8273839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8274184Z res = mod(**inputs) 2025-12-04T09:50:51.8274581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8275043Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8275497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8275916Z outputs = layer_module( 2025-12-04T09:50:51.8276347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8276775Z outputs = self.rel_attn( 2025-12-04T09:50:51.8277185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8277617Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8278056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8278556Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8278752Z 2025-12-04T09:50:51.8278866Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8279260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8279611Z res = mod(**inputs) 2025-12-04T09:50:51.8280010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8280442Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8280972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8281403Z outputs = layer_module( 2025-12-04T09:50:51.8281811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8282241Z outputs = self.rel_attn( 2025-12-04T09:50:51.8282666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8283110Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8283566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8284102Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8284341Z 2025-12-04T09:50:51.8284458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8284851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8285199Z res = mod(**inputs) 2025-12-04T09:50:51.8285598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8286037Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8286455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8286869Z outputs = layer_module( 2025-12-04T09:50:51.8287267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8287683Z outputs = self.rel_attn( 2025-12-04T09:50:51.8288091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8288549Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8289019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8289515Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8289698Z 2025-12-04T09:50:51.8289815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8290213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8290594Z res = mod(**inputs) 2025-12-04T09:50:51.8290986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8291425Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8291885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8292311Z outputs = layer_module( 2025-12-04T09:50:51.8292709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8293288Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8293873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8294312Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8294736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8295172Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8295594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8296018Z output = self.layer_1(output) 2025-12-04T09:50:51.8296161Z 2025-12-04T09:50:51.8296275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8296675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8297048Z res = mod(**inputs) 2025-12-04T09:50:51.8297462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8297901Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8298341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8298762Z outputs = layer_module( 2025-12-04T09:50:51.8299200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8299800Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8300402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8300838Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8301267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8301696Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8302114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8302555Z output = self.activation_function(output) 2025-12-04T09:50:51.8302955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8303340Z return self.act(input) 2025-12-04T09:50:51.8303485Z 2025-12-04T09:50:51.8303607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8304008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8304370Z res = mod(**inputs) 2025-12-04T09:50:51.8304770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8305210Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8305651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8306102Z outputs = layer_module( 2025-12-04T09:50:51.8306502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8307085Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8307675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8308134Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8308552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8308993Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8309433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8309859Z output = self.layer_2(output) 2025-12-04T09:50:51.8309992Z 2025-12-04T09:50:51.8310104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8310494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8310845Z res = mod(**inputs) 2025-12-04T09:50:51.8311226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8311672Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8312126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8312556Z outputs = layer_module( 2025-12-04T09:50:51.8312947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8313386Z outputs = self.rel_attn( 2025-12-04T09:50:51.8313809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8314263Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8314430Z 2025-12-04T09:50:51.8314546Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8314972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8315352Z res = mod(**inputs) 2025-12-04T09:50:51.8315736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8316169Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8316595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8317022Z outputs = layer_module( 2025-12-04T09:50:51.8317413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8317842Z outputs = self.rel_attn( 2025-12-04T09:50:51.8318267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8318723Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8318912Z 2025-12-04T09:50:51.8319030Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8319415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8319759Z res = mod(**inputs) 2025-12-04T09:50:51.8320136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8320653Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8321103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8321555Z outputs = layer_module( 2025-12-04T09:50:51.8321956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8322388Z outputs = self.rel_attn( 2025-12-04T09:50:51.8322791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8323207Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8323655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8324173Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8324382Z 2025-12-04T09:50:51.8324507Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8324902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8325263Z res = mod(**inputs) 2025-12-04T09:50:51.8325668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8326114Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8326549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8326977Z outputs = layer_module( 2025-12-04T09:50:51.8327383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8327803Z outputs = self.rel_attn( 2025-12-04T09:50:51.8328213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8328713Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8328920Z 2025-12-04T09:50:51.8329044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8329432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8329786Z res = mod(**inputs) 2025-12-04T09:50:51.8330074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8330222Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8330508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8330592Z outputs = layer_module( 2025-12-04T09:50:51.8330873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8330951Z outputs = self.rel_attn( 2025-12-04T09:50:51.8331242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8331327Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8331628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8331779Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8331807Z 2025-12-04T09:50:51.8331928Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8332156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8332228Z res = mod(**inputs) 2025-12-04T09:50:51.8332531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8332633Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8332936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8333040Z outputs = layer_module( 2025-12-04T09:50:51.8333327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8333403Z outputs = self.rel_attn( 2025-12-04T09:50:51.8333710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8333827Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8333831Z 2025-12-04T09:50:51.8333943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8334170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8334241Z res = mod(**inputs) 2025-12-04T09:50:51.8334544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8334639Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8334920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8335002Z outputs = layer_module( 2025-12-04T09:50:51.8335304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8335391Z outputs = self.rel_attn( 2025-12-04T09:50:51.8335673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8335753Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8336061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8336202Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8336206Z 2025-12-04T09:50:51.8336321Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8336546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8336616Z res = mod(**inputs) 2025-12-04T09:50:51.8336908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8337036Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8337341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8337425Z outputs = layer_module( 2025-12-04T09:50:51.8337728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8337811Z outputs = self.rel_attn( 2025-12-04T09:50:51.8338091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8338196Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8338506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8338631Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8338651Z 2025-12-04T09:50:51.8338766Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8338991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8339064Z res = mod(**inputs) 2025-12-04T09:50:51.8339349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8339442Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8339721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8339825Z outputs = layer_module( 2025-12-04T09:50:51.8340102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8340177Z outputs = self.rel_attn( 2025-12-04T09:50:51.8340465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8340564Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8340871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8340993Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8340997Z 2025-12-04T09:50:51.8341108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8341329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8341402Z res = mod(**inputs) 2025-12-04T09:50:51.8341687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8341779Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8342061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8342147Z outputs = layer_module( 2025-12-04T09:50:51.8342425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8342665Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8342956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8343047Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8343344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8343423Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8343694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8343802Z output = self.layer_1(output) 2025-12-04T09:50:51.8343822Z 2025-12-04T09:50:51.8343937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8344161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8344235Z res = mod(**inputs) 2025-12-04T09:50:51.8344535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8344633Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8344905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8344983Z outputs = layer_module( 2025-12-04T09:50:51.8345272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8345523Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8345824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8345914Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8346195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8346284Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8346563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8346688Z output = self.activation_function(output) 2025-12-04T09:50:51.8346931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8347010Z return self.act(input) 2025-12-04T09:50:51.8347014Z 2025-12-04T09:50:51.8347322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8347555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8347628Z res = mod(**inputs) 2025-12-04T09:50:51.8347925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8348017Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8348304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8348380Z outputs = layer_module( 2025-12-04T09:50:51.8348656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8348893Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8349179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8349274Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8349551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8349631Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8349914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8349994Z output = self.layer_2(output) 2025-12-04T09:50:51.8350001Z 2025-12-04T09:50:51.8350124Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8350341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8350410Z res = mod(**inputs) 2025-12-04T09:50:51.8350714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8350899Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8351176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8351257Z outputs = layer_module( 2025-12-04T09:50:51.8351529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8351613Z outputs = self.rel_attn( 2025-12-04T09:50:51.8351882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8351991Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8351995Z 2025-12-04T09:50:51.8352111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8352323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8352419Z res = mod(**inputs) 2025-12-04T09:50:51.8352709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8352800Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8353080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8353154Z outputs = layer_module( 2025-12-04T09:50:51.8353426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8353539Z outputs = self.rel_attn( 2025-12-04T09:50:51.8353811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8353927Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8353931Z 2025-12-04T09:50:51.8354041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8354258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8354335Z res = mod(**inputs) 2025-12-04T09:50:51.8354608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8354697Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8354979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8355052Z outputs = layer_module( 2025-12-04T09:50:51.8355328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8355403Z outputs = self.rel_attn( 2025-12-04T09:50:51.8355703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8355792Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8356087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8356237Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8356241Z 2025-12-04T09:50:51.8356351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8356564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8356641Z res = mod(**inputs) 2025-12-04T09:50:51.8356919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8357008Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8357288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8357362Z outputs = layer_module( 2025-12-04T09:50:51.8357693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8357769Z outputs = self.rel_attn( 2025-12-04T09:50:51.8358041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8358194Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8358198Z 2025-12-04T09:50:51.8358306Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8358526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8358594Z res = mod(**inputs) 2025-12-04T09:50:51.8358870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8358985Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8359268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8359342Z outputs = layer_module( 2025-12-04T09:50:51.8359622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8359698Z outputs = self.rel_attn( 2025-12-04T09:50:51.8359976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8360087Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8360382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8360586Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8360593Z 2025-12-04T09:50:51.8360712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8360942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8361013Z res = mod(**inputs) 2025-12-04T09:50:51.8361298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8361398Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8361693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8361779Z outputs = layer_module( 2025-12-04T09:50:51.8362061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8362136Z outputs = self.rel_attn( 2025-12-04T09:50:51.8362415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8362527Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8362532Z 2025-12-04T09:50:51.8362642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8362861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8362932Z res = mod(**inputs) 2025-12-04T09:50:51.8363208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8363306Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8363581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8363664Z outputs = layer_module( 2025-12-04T09:50:51.8363937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8364015Z outputs = self.rel_attn( 2025-12-04T09:50:51.8364344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8364425Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8364723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8364858Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8364862Z 2025-12-04T09:50:51.8364971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8365190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8365258Z res = mod(**inputs) 2025-12-04T09:50:51.8365531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8365647Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8365924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8366005Z outputs = layer_module( 2025-12-04T09:50:51.8366278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8366354Z outputs = self.rel_attn( 2025-12-04T09:50:51.8366642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8366740Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8367077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8367199Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8367203Z 2025-12-04T09:50:51.8367313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8367536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8367608Z res = mod(**inputs) 2025-12-04T09:50:51.8367884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8367981Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8368253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8368334Z outputs = layer_module( 2025-12-04T09:50:51.8368606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8368679Z outputs = self.rel_attn( 2025-12-04T09:50:51.8368959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8369059Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8369355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8369485Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8369489Z 2025-12-04T09:50:51.8369600Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8369824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8369895Z res = mod(**inputs) 2025-12-04T09:50:51.8370182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8370281Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8370554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8370633Z outputs = layer_module( 2025-12-04T09:50:51.8370984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8371217Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8371527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8371612Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8371893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8371972Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8372243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8372328Z output = self.layer_1(output) 2025-12-04T09:50:51.8372351Z 2025-12-04T09:50:51.8372462Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8372674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8372752Z res = mod(**inputs) 2025-12-04T09:50:51.8373026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8373120Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8373389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8373483Z outputs = layer_module( 2025-12-04T09:50:51.8373769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8373993Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8374292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8374381Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8374661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8374743Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8375061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8375159Z output = self.activation_function(output) 2025-12-04T09:50:51.8375404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8375479Z return self.act(input) 2025-12-04T09:50:51.8375482Z 2025-12-04T09:50:51.8375601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8375820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8375891Z res = mod(**inputs) 2025-12-04T09:50:51.8376181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8376270Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8376583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8376663Z outputs = layer_module( 2025-12-04T09:50:51.8376982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8377218Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8377510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8377614Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8377921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8377999Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8378274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8378353Z output = self.layer_2(output) 2025-12-04T09:50:51.8378357Z 2025-12-04T09:50:51.8378468Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8378687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8378757Z res = mod(**inputs) 2025-12-04T09:50:51.8379029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8379145Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8379422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8379502Z outputs = layer_module( 2025-12-04T09:50:51.8379775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8379852Z outputs = self.rel_attn( 2025-12-04T09:50:51.8380129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8380255Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8380259Z 2025-12-04T09:50:51.8380374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8380586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8380654Z res = mod(**inputs) 2025-12-04T09:50:51.8380938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8381029Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8381300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8381389Z outputs = layer_module( 2025-12-04T09:50:51.8381658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8381740Z outputs = self.rel_attn( 2025-12-04T09:50:51.8382012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8382122Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8382125Z 2025-12-04T09:50:51.8382242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8382454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8382532Z res = mod(**inputs) 2025-12-04T09:50:51.8382802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8382889Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8383167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8383240Z outputs = layer_module( 2025-12-04T09:50:51.8383509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8383593Z outputs = self.rel_attn( 2025-12-04T09:50:51.8383861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8383948Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8384265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8384426Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8384430Z 2025-12-04T09:50:51.8384547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8384757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8384835Z res = mod(**inputs) 2025-12-04T09:50:51.8385109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8385201Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8385482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8385555Z outputs = layer_module( 2025-12-04T09:50:51.8385862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8385948Z outputs = self.rel_attn( 2025-12-04T09:50:51.8386225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8386374Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8386378Z 2025-12-04T09:50:51.8386487Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8386697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8386793Z res = mod(**inputs) 2025-12-04T09:50:51.8387071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8387167Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8387445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8387519Z outputs = layer_module( 2025-12-04T09:50:51.8387800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8387873Z outputs = self.rel_attn( 2025-12-04T09:50:51.8388147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8388234Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8388527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8388670Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8388674Z 2025-12-04T09:50:51.8388784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8388998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8389076Z res = mod(**inputs) 2025-12-04T09:50:51.8389352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8389446Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8389719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8389792Z outputs = layer_module( 2025-12-04T09:50:51.8390070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8390146Z outputs = self.rel_attn( 2025-12-04T09:50:51.8390418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8390534Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8390538Z 2025-12-04T09:50:51.8390681Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8390902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8390972Z res = mod(**inputs) 2025-12-04T09:50:51.8391246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8391342Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8391614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8391688Z outputs = layer_module( 2025-12-04T09:50:51.8391965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8392039Z outputs = self.rel_attn( 2025-12-04T09:50:51.8392331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8392414Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8392704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8392848Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8392852Z 2025-12-04T09:50:51.8392961Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8393178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8393268Z res = mod(**inputs) 2025-12-04T09:50:51.8393548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8393653Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8393920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8393991Z outputs = layer_module( 2025-12-04T09:50:51.8394263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8394333Z outputs = self.rel_attn( 2025-12-04T09:50:51.8394603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8394695Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8394981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8395102Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8395107Z 2025-12-04T09:50:51.8395212Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8395424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8395492Z res = mod(**inputs) 2025-12-04T09:50:51.8395755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8395849Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8396113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8396182Z outputs = layer_module( 2025-12-04T09:50:51.8396452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8396523Z outputs = self.rel_attn( 2025-12-04T09:50:51.8396805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8396899Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8397221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8397366Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8397370Z 2025-12-04T09:50:51.8397478Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8397695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8397763Z res = mod(**inputs) 2025-12-04T09:50:51.8398034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8398132Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8398401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8398472Z outputs = layer_module( 2025-12-04T09:50:51.8398801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8399025Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8399315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8399399Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8399671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8399775Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8400054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8400138Z output = self.layer_1(output) 2025-12-04T09:50:51.8400142Z 2025-12-04T09:50:51.8400256Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8400482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8400643Z res = mod(**inputs) 2025-12-04T09:50:51.8400932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8401024Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8401324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8401403Z outputs = layer_module( 2025-12-04T09:50:51.8401706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8401940Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8402227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8402321Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8402594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8402681Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8402962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8403058Z output = self.activation_function(output) 2025-12-04T09:50:51.8403310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8403385Z return self.act(input) 2025-12-04T09:50:51.8403389Z 2025-12-04T09:50:51.8403490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8403697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8403784Z res = mod(**inputs) 2025-12-04T09:50:51.8404078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8404170Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8404450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8404533Z outputs = layer_module( 2025-12-04T09:50:51.8404817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8405044Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8405321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8405423Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8405715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8405793Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8406077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8406161Z output = self.layer_2(output) 2025-12-04T09:50:51.8406165Z 2025-12-04T09:50:51.8406273Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8406496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8406581Z res = mod(**inputs) 2025-12-04T09:50:51.8406870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8406967Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8407259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8407344Z outputs = layer_module( 2025-12-04T09:50:51.8407644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8407722Z outputs = self.rel_attn( 2025-12-04T09:50:51.8408024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8408133Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8408138Z 2025-12-04T09:50:51.8408248Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8408471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8408541Z res = mod(**inputs) 2025-12-04T09:50:51.8408838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8408932Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8409223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8409303Z outputs = layer_module( 2025-12-04T09:50:51.8409637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8409712Z outputs = self.rel_attn( 2025-12-04T09:50:51.8410008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8410118Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8410122Z 2025-12-04T09:50:51.8410239Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8410455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8410546Z res = mod(**inputs) 2025-12-04T09:50:51.8410909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8411000Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8411278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8411351Z outputs = layer_module( 2025-12-04T09:50:51.8411621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8411706Z outputs = self.rel_attn( 2025-12-04T09:50:51.8411975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8412055Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8412373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8412520Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8412524Z 2025-12-04T09:50:51.8412641Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8412850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8412919Z res = mod(**inputs) 2025-12-04T09:50:51.8413202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8413310Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8413590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8413662Z outputs = layer_module( 2025-12-04T09:50:51.8413938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8414020Z outputs = self.rel_attn( 2025-12-04T09:50:51.8414297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8414442Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8414453Z 2025-12-04T09:50:51.8414563Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8414774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8414853Z res = mod(**inputs) 2025-12-04T09:50:51.8415127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8415215Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8415500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8415575Z outputs = layer_module( 2025-12-04T09:50:51.8415854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8415928Z outputs = self.rel_attn( 2025-12-04T09:50:51.8416200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8416286Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8416580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8416722Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8416726Z 2025-12-04T09:50:51.8416844Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8417057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8417150Z res = mod(**inputs) 2025-12-04T09:50:51.8417442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8417532Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8417812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8417884Z outputs = layer_module( 2025-12-04T09:50:51.8418159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8418236Z outputs = self.rel_attn( 2025-12-04T09:50:51.8418503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8418617Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8418621Z 2025-12-04T09:50:51.8418748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8418962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8419041Z res = mod(**inputs) 2025-12-04T09:50:51.8419326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8419424Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8419702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8419795Z outputs = layer_module( 2025-12-04T09:50:51.8420097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8420172Z outputs = self.rel_attn( 2025-12-04T09:50:51.8420475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8420565Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8420871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8421016Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8421019Z 2025-12-04T09:50:51.8421131Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8421351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8421429Z res = mod(**inputs) 2025-12-04T09:50:51.8421719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8421816Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8422103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8422178Z outputs = layer_module( 2025-12-04T09:50:51.8422473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8422547Z outputs = self.rel_attn( 2025-12-04T09:50:51.8422827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8422933Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8423240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8423374Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8423378Z 2025-12-04T09:50:51.8423488Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8423708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8423788Z res = mod(**inputs) 2025-12-04T09:50:51.8424107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8424207Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8424489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8424564Z outputs = layer_module( 2025-12-04T09:50:51.8424851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8424929Z outputs = self.rel_attn( 2025-12-04T09:50:51.8425208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8425314Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8425633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8425769Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8425773Z 2025-12-04T09:50:51.8425885Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8426104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8426183Z res = mod(**inputs) 2025-12-04T09:50:51.8426464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8426587Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8426867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8426942Z outputs = layer_module( 2025-12-04T09:50:51.8427228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8427458Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8427750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8427846Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8428127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8428213Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8428493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8428574Z output = self.layer_1(output) 2025-12-04T09:50:51.8428578Z 2025-12-04T09:50:51.8428700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8428916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8428998Z res = mod(**inputs) 2025-12-04T09:50:51.8429275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8429366Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8429649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8429724Z outputs = layer_module( 2025-12-04T09:50:51.8430002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8430236Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8430522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8430636Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8430936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8431016Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8431302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8431400Z output = self.activation_function(output) 2025-12-04T09:50:51.8431643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8431723Z return self.act(input) 2025-12-04T09:50:51.8431727Z 2025-12-04T09:50:51.8431840Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8432063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8432134Z res = mod(**inputs) 2025-12-04T09:50:51.8432436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8432539Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8432820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8432901Z outputs = layer_module( 2025-12-04T09:50:51.8433177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8433403Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8433719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8433806Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8434102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8434184Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8434463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8434550Z output = self.layer_2(output) 2025-12-04T09:50:51.8434554Z 2025-12-04T09:50:51.8434666Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8434887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8434968Z res = mod(**inputs) 2025-12-04T09:50:51.8435250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8435345Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8435625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8435703Z outputs = layer_module( 2025-12-04T09:50:51.8435994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8436072Z outputs = self.rel_attn( 2025-12-04T09:50:51.8436350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8436466Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8436470Z 2025-12-04T09:50:51.8436581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8436805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8436876Z res = mod(**inputs) 2025-12-04T09:50:51.8437178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8437279Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8437608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8437692Z outputs = layer_module( 2025-12-04T09:50:51.8437973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8438049Z outputs = self.rel_attn( 2025-12-04T09:50:51.8438353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8438467Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8438471Z 2025-12-04T09:50:51.8438586Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8438802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8438873Z res = mod(**inputs) 2025-12-04T09:50:51.8439200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8439294Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8439573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8439657Z outputs = layer_module( 2025-12-04T09:50:51.8439934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8440016Z outputs = self.rel_attn( 2025-12-04T09:50:51.8440315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8440397Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8440967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8441121Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8441127Z 2025-12-04T09:50:51.8441241Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8441468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8441541Z res = mod(**inputs) 2025-12-04T09:50:51.8441828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8441921Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8442204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8442288Z outputs = layer_module( 2025-12-04T09:50:51.8442570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8442656Z outputs = self.rel_attn( 2025-12-04T09:50:51.8442939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8443088Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8443092Z 2025-12-04T09:50:51.8443212Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8443428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8443499Z res = mod(**inputs) 2025-12-04T09:50:51.8443787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8443880Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8444168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8444244Z outputs = layer_module( 2025-12-04T09:50:51.8444547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8444650Z outputs = self.rel_attn( 2025-12-04T09:50:51.8444935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8445024Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8445337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8445483Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8445487Z 2025-12-04T09:50:51.8445613Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8445828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8445896Z res = mod(**inputs) 2025-12-04T09:50:51.8446201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8446292Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8446575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8446648Z outputs = layer_module( 2025-12-04T09:50:51.8446927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8447008Z outputs = self.rel_attn( 2025-12-04T09:50:51.8447494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8447603Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8447615Z 2025-12-04T09:50:51.8447724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8447940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8448019Z res = mod(**inputs) 2025-12-04T09:50:51.8448303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8448392Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8448674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8448746Z outputs = layer_module( 2025-12-04T09:50:51.8449028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8449104Z outputs = self.rel_attn( 2025-12-04T09:50:51.8449376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8449466Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8449757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8449893Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8449905Z 2025-12-04T09:50:51.8450015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8450226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8450303Z res = mod(**inputs) 2025-12-04T09:50:51.8450577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8450667Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8450947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8451020Z outputs = layer_module( 2025-12-04T09:50:51.8451354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8451455Z outputs = self.rel_attn( 2025-12-04T09:50:51.8451735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8451839Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8452146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8452269Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8452280Z 2025-12-04T09:50:51.8452390Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8452606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8452681Z res = mod(**inputs) 2025-12-04T09:50:51.8452993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8453085Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8453366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8453437Z outputs = layer_module( 2025-12-04T09:50:51.8453715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8453789Z outputs = self.rel_attn( 2025-12-04T09:50:51.8454094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8454192Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8454473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8454587Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8454599Z 2025-12-04T09:50:51.8454702Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8454900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8454973Z res = mod(**inputs) 2025-12-04T09:50:51.8455231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8455313Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8455576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8455643Z outputs = layer_module( 2025-12-04T09:50:51.8455905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8456128Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8456413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8456503Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8456774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8456850Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8457123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8457203Z output = self.layer_1(output) 2025-12-04T09:50:51.8457207Z 2025-12-04T09:50:51.8457322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8457532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8457600Z res = mod(**inputs) 2025-12-04T09:50:51.8457921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8458009Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8458286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8458368Z outputs = layer_module( 2025-12-04T09:50:51.8458625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8458847Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8459123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8459205Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8459506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8459586Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8459864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8459957Z output = self.activation_function(output) 2025-12-04T09:50:51.8460187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8460270Z return self.act(input) 2025-12-04T09:50:51.8460302Z 2025-12-04T09:50:51.8460415Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8460631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8460701Z res = mod(**inputs) 2025-12-04T09:50:51.8460974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8461071Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8461343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8461416Z outputs = layer_module( 2025-12-04T09:50:51.8461695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8461915Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8462206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8462290Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8462565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8462653Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8462927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8463012Z output = self.layer_2(output) 2025-12-04T09:50:51.8463017Z 2025-12-04T09:50:51.8463125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8463337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8463413Z res = mod(**inputs) 2025-12-04T09:50:51.8463688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8463778Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8464095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8464168Z outputs = layer_module( 2025-12-04T09:50:51.8464495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8464595Z outputs = self.rel_attn( 2025-12-04T09:50:51.8464863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8464977Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8464981Z 2025-12-04T09:50:51.8465091Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8465356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8465440Z res = mod(**inputs) 2025-12-04T09:50:51.8465730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8465824Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8466136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8466211Z outputs = layer_module( 2025-12-04T09:50:51.8466514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8466588Z outputs = self.rel_attn( 2025-12-04T09:50:51.8466891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8466998Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8467019Z 2025-12-04T09:50:51.8467131Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8467350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8467420Z res = mod(**inputs) 2025-12-04T09:50:51.8467708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8467804Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8468110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8468191Z outputs = layer_module( 2025-12-04T09:50:51.8468489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8468562Z outputs = self.rel_attn( 2025-12-04T09:50:51.8468872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8468953Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8469253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8469395Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8469401Z 2025-12-04T09:50:51.8469513Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8469746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8469816Z res = mod(**inputs) 2025-12-04T09:50:51.8470102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8470197Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8470482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8470563Z outputs = layer_module( 2025-12-04T09:50:51.8470857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8470930Z outputs = self.rel_attn( 2025-12-04T09:50:51.8471240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8471403Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8471406Z 2025-12-04T09:50:51.8471523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8471777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8471858Z res = mod(**inputs) 2025-12-04T09:50:51.8472149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8472238Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8472519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8472597Z outputs = layer_module( 2025-12-04T09:50:51.8472911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8472998Z outputs = self.rel_attn( 2025-12-04T09:50:51.8473299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8473378Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8473680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8473820Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8473844Z 2025-12-04T09:50:51.8473960Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8474167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8474236Z res = mod(**inputs) 2025-12-04T09:50:51.8474515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8474605Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8474879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8474960Z outputs = layer_module( 2025-12-04T09:50:51.8475229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8475312Z outputs = self.rel_attn( 2025-12-04T09:50:51.8475581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8475691Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8475694Z 2025-12-04T09:50:51.8475810Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8476023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8476102Z res = mod(**inputs) 2025-12-04T09:50:51.8476374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8476462Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8476745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8476814Z outputs = layer_module( 2025-12-04T09:50:51.8477066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8477144Z outputs = self.rel_attn( 2025-12-04T09:50:51.8477409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8477494Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8477809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8477961Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8477965Z 2025-12-04T09:50:51.8478081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8478292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8478361Z res = mod(**inputs) 2025-12-04T09:50:51.8478643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8478735Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8479016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8479088Z outputs = layer_module( 2025-12-04T09:50:51.8479408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8479496Z outputs = self.rel_attn( 2025-12-04T09:50:51.8479774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8479879Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8480182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8480305Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8480326Z 2025-12-04T09:50:51.8480449Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8480734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8480809Z res = mod(**inputs) 2025-12-04T09:50:51.8481108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8481203Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8481492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8481568Z outputs = layer_module( 2025-12-04T09:50:51.8481872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8481959Z outputs = self.rel_attn( 2025-12-04T09:50:51.8482230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8482329Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8482612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8482725Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8482729Z 2025-12-04T09:50:51.8482845Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8483045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8483112Z res = mod(**inputs) 2025-12-04T09:50:51.8483380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8483465Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8483732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8483803Z outputs = layer_module( 2025-12-04T09:50:51.8484061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8484282Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8484588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8484679Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8484942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8485016Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8485280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8485356Z output = self.layer_1(output) 2025-12-04T09:50:51.8485360Z 2025-12-04T09:50:51.8485465Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8485674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8485740Z res = mod(**inputs) 2025-12-04T09:50:51.8486030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8486119Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8486376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8486453Z outputs = layer_module( 2025-12-04T09:50:51.8486707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8486921Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8487206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8487286Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8487554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8487628Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8487886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8487983Z output = self.activation_function(output) 2025-12-04T09:50:51.8488203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8488283Z return self.act(input) 2025-12-04T09:50:51.8488286Z 2025-12-04T09:50:51.8488388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8488589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8488661Z res = mod(**inputs) 2025-12-04T09:50:51.8488918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8489010Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8489267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8489338Z outputs = layer_module( 2025-12-04T09:50:51.8489596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8489810Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8490069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8490154Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8490403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8490480Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8490747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8490836Z output = self.layer_2(output) 2025-12-04T09:50:51.8490839Z 2025-12-04T09:50:51.8490950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8491144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8491215Z res = mod(**inputs) 2025-12-04T09:50:51.8491469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8491553Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8491810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8491877Z outputs = layer_module( 2025-12-04T09:50:51.8492145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8492226Z outputs = self.rel_attn( 2025-12-04T09:50:51.8492477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8492583Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8492586Z 2025-12-04T09:50:51.8492687Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8492881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8492971Z res = mod(**inputs) 2025-12-04T09:50:51.8493228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8493319Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8493579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8493650Z outputs = layer_module( 2025-12-04T09:50:51.8493917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8493986Z outputs = self.rel_attn( 2025-12-04T09:50:51.8494238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8494346Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8494350Z 2025-12-04T09:50:51.8494450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8494657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8494722Z res = mod(**inputs) 2025-12-04T09:50:51.8494980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8495073Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8495331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8495400Z outputs = layer_module( 2025-12-04T09:50:51.8495662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8495733Z outputs = self.rel_attn( 2025-12-04T09:50:51.8495994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8496071Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8496344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8496486Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8496490Z 2025-12-04T09:50:51.8496618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8496841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8496907Z res = mod(**inputs) 2025-12-04T09:50:51.8497166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8497256Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8497509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8497578Z outputs = layer_module( 2025-12-04T09:50:51.8497842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8497911Z outputs = self.rel_attn( 2025-12-04T09:50:51.8498189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8498330Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8498334Z 2025-12-04T09:50:51.8498437Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8498643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8498709Z res = mod(**inputs) 2025-12-04T09:50:51.8498974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8499060Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8499339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8499416Z outputs = layer_module( 2025-12-04T09:50:51.8499687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8499757Z outputs = self.rel_attn( 2025-12-04T09:50:51.8500017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8500090Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8500370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8500503Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8500506Z 2025-12-04T09:50:51.8500609Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8500821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8500885Z res = mod(**inputs) 2025-12-04T09:50:51.8501153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8501239Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8501499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8501576Z outputs = layer_module( 2025-12-04T09:50:51.8501833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8501902Z outputs = self.rel_attn( 2025-12-04T09:50:51.8502164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8502268Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8502271Z 2025-12-04T09:50:51.8502381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8502581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8502648Z res = mod(**inputs) 2025-12-04T09:50:51.8502939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8503069Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8503336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8503407Z outputs = layer_module( 2025-12-04T09:50:51.8503679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8503759Z outputs = self.rel_attn( 2025-12-04T09:50:51.8504037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8504111Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8504415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8504548Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8504552Z 2025-12-04T09:50:51.8504664Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8504875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8504943Z res = mod(**inputs) 2025-12-04T09:50:51.8505220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8505316Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8505582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8505656Z outputs = layer_module( 2025-12-04T09:50:51.8505901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8505977Z outputs = self.rel_attn( 2025-12-04T09:50:51.8506229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8506316Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8506592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8506701Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8506705Z 2025-12-04T09:50:51.8506811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8507004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8507069Z res = mod(**inputs) 2025-12-04T09:50:51.8507326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8507409Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8507665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8507738Z outputs = layer_module( 2025-12-04T09:50:51.8507986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8508062Z outputs = self.rel_attn( 2025-12-04T09:50:51.8508309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8508398Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8508674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8508784Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8508787Z 2025-12-04T09:50:51.8508896Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8509133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8509199Z res = mod(**inputs) 2025-12-04T09:50:51.8509454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8509536Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8509788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8509864Z outputs = layer_module( 2025-12-04T09:50:51.8510112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8510323Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8510600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8510682Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8510941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8511013Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8511270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8511342Z output = self.layer_1(output) 2025-12-04T09:50:51.8511361Z 2025-12-04T09:50:51.8511463Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8511665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8511730Z res = mod(**inputs) 2025-12-04T09:50:51.8511985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8512078Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8512327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8512410Z outputs = layer_module( 2025-12-04T09:50:51.8512665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8512872Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8513145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8513224Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8513491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8513569Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8513825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8513926Z output = self.activation_function(output) 2025-12-04T09:50:51.8514157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8514233Z return self.act(input) 2025-12-04T09:50:51.8514243Z 2025-12-04T09:50:51.8514353Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8514572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8514646Z res = mod(**inputs) 2025-12-04T09:50:51.8514903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8514985Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8515283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8515374Z outputs = layer_module( 2025-12-04T09:50:51.8515656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8515877Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8516165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8516258Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8516539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8516614Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8516915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8516998Z output = self.layer_2(output) 2025-12-04T09:50:51.8517002Z 2025-12-04T09:50:51.8517119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8517332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8517400Z res = mod(**inputs) 2025-12-04T09:50:51.8517680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8517769Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8518066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8518139Z outputs = layer_module( 2025-12-04T09:50:51.8518408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8518491Z outputs = self.rel_attn( 2025-12-04T09:50:51.8518766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8518874Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8518877Z 2025-12-04T09:50:51.8518996Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8519210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8519285Z res = mod(**inputs) 2025-12-04T09:50:51.8519571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8519660Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8519948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8520023Z outputs = layer_module( 2025-12-04T09:50:51.8520315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8520392Z outputs = self.rel_attn( 2025-12-04T09:50:51.8520770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8520895Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8520901Z 2025-12-04T09:50:51.8521014Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8521236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8521317Z res = mod(**inputs) 2025-12-04T09:50:51.8521597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8521709Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8522351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8522458Z outputs = layer_module( 2025-12-04T09:50:51.8522739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8522816Z outputs = self.rel_attn( 2025-12-04T09:50:51.8523127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8523216Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8523514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8523662Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8523666Z 2025-12-04T09:50:51.8523774Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8524005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8524087Z res = mod(**inputs) 2025-12-04T09:50:51.8524364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8524463Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8524741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8524814Z outputs = layer_module( 2025-12-04T09:50:51.8525117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8525193Z outputs = self.rel_attn( 2025-12-04T09:50:51.8525463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8525618Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8525623Z 2025-12-04T09:50:51.8525734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8525950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8526021Z res = mod(**inputs) 2025-12-04T09:50:51.8526294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8526389Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8526660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8526741Z outputs = layer_module( 2025-12-04T09:50:51.8527011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8527084Z outputs = self.rel_attn( 2025-12-04T09:50:51.8527364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8527443Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8527734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8527879Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8527883Z 2025-12-04T09:50:51.8527991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8528210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8528279Z res = mod(**inputs) 2025-12-04T09:50:51.8528553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8528649Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8528942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8529041Z outputs = layer_module( 2025-12-04T09:50:51.8529309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8529383Z outputs = self.rel_attn( 2025-12-04T09:50:51.8529657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8529765Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8529770Z 2025-12-04T09:50:51.8529878Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8530096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8530166Z res = mod(**inputs) 2025-12-04T09:50:51.8530464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8530558Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8530836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8530916Z outputs = layer_module( 2025-12-04T09:50:51.8531186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8531266Z outputs = self.rel_attn( 2025-12-04T09:50:51.8531536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8531636Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8531935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8532072Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8532079Z 2025-12-04T09:50:51.8532188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8532407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8532476Z res = mod(**inputs) 2025-12-04T09:50:51.8532758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8532846Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8533125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8533208Z outputs = layer_module( 2025-12-04T09:50:51.8533481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8533555Z outputs = self.rel_attn( 2025-12-04T09:50:51.8533839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8533937Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8534239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8534358Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8534362Z 2025-12-04T09:50:51.8534470Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8534687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8534759Z res = mod(**inputs) 2025-12-04T09:50:51.8535040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8535129Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8535433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8535533Z outputs = layer_module( 2025-12-04T09:50:51.8535806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8535879Z outputs = self.rel_attn( 2025-12-04T09:50:51.8536161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8536255Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8536556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8536675Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8536678Z 2025-12-04T09:50:51.8536787Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8537024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8537095Z res = mod(**inputs) 2025-12-04T09:50:51.8537379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8537466Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8537741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8537820Z outputs = layer_module( 2025-12-04T09:50:51.8538111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8538334Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8538629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8538717Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8538996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8539073Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8539342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8539427Z output = self.layer_1(output) 2025-12-04T09:50:51.8539431Z 2025-12-04T09:50:51.8539541Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8539760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8539830Z res = mod(**inputs) 2025-12-04T09:50:51.8540100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8540198Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8540473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8540546Z outputs = layer_module( 2025-12-04T09:50:51.8540824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8541043Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8541332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8541417Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8541687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8541773Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8542057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8542178Z output = self.activation_function(output) 2025-12-04T09:50:51.8542409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8542485Z return self.act(input) 2025-12-04T09:50:51.8542489Z 2025-12-04T09:50:51.8542608Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8542820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8542893Z res = mod(**inputs) 2025-12-04T09:50:51.8543179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8543267Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8543569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8543645Z outputs = layer_module( 2025-12-04T09:50:51.8543916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8544144Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8544427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8544516Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8544817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8544893Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8545169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8545249Z output = self.layer_2(output) 2025-12-04T09:50:51.8545254Z 2025-12-04T09:50:51.8545365Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8545583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8545653Z res = mod(**inputs) 2025-12-04T09:50:51.8545931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8546020Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8546292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8546373Z outputs = layer_module( 2025-12-04T09:50:51.8546640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8546722Z outputs = self.rel_attn( 2025-12-04T09:50:51.8547004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8547318Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8547325Z 2025-12-04T09:50:51.8547446Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8547658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8547729Z res = mod(**inputs) 2025-12-04T09:50:51.8548012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8548103Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8548400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8548473Z outputs = layer_module( 2025-12-04T09:50:51.8548797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8548909Z outputs = self.rel_attn( 2025-12-04T09:50:51.8549200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8549317Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8549321Z 2025-12-04T09:50:51.8549433Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8549650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8549735Z res = mod(**inputs) 2025-12-04T09:50:51.8550018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8550111Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8550440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8550516Z outputs = layer_module( 2025-12-04T09:50:51.8550806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8550879Z outputs = self.rel_attn( 2025-12-04T09:50:51.8551148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8551233Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8551506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8551671Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8551682Z 2025-12-04T09:50:51.8551787Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8551986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8552065Z res = mod(**inputs) 2025-12-04T09:50:51.8552326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8552415Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8552683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8552755Z outputs = layer_module( 2025-12-04T09:50:51.8553024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8553099Z outputs = self.rel_attn( 2025-12-04T09:50:51.8553361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8553509Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8553514Z 2025-12-04T09:50:51.8553621Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8553820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8553896Z res = mod(**inputs) 2025-12-04T09:50:51.8554158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8554250Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8554510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8554582Z outputs = layer_module( 2025-12-04T09:50:51.8554850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8554921Z outputs = self.rel_attn( 2025-12-04T09:50:51.8555226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8555322Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8555614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8555759Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8555763Z 2025-12-04T09:50:51.8555871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8556082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8556158Z res = mod(**inputs) 2025-12-04T09:50:51.8556428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8556524Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8556811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8556887Z outputs = layer_module( 2025-12-04T09:50:51.8557165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8557239Z outputs = self.rel_attn( 2025-12-04T09:50:51.8557515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8557623Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8557626Z 2025-12-04T09:50:51.8557762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8557980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8558050Z res = mod(**inputs) 2025-12-04T09:50:51.8558328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8558428Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8558705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8558783Z outputs = layer_module( 2025-12-04T09:50:51.8559056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8559131Z outputs = self.rel_attn( 2025-12-04T09:50:51.8559411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8559490Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8559783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8559923Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8559927Z 2025-12-04T09:50:51.8560039Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8560261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8560329Z res = mod(**inputs) 2025-12-04T09:50:51.8560667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8560770Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8561054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8561140Z outputs = layer_module( 2025-12-04T09:50:51.8561418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8561495Z outputs = self.rel_attn( 2025-12-04T09:50:51.8561784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8561923Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8562232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8562369Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8562373Z 2025-12-04T09:50:51.8562488Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8562719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8562793Z res = mod(**inputs) 2025-12-04T09:50:51.8563085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8563180Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8563458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8563541Z outputs = layer_module( 2025-12-04T09:50:51.8563800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8563872Z outputs = self.rel_attn( 2025-12-04T09:50:51.8564134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8564232Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8564534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8564680Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8564684Z 2025-12-04T09:50:51.8564796Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8565022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8565096Z res = mod(**inputs) 2025-12-04T09:50:51.8565380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8565478Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8565779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8565863Z outputs = layer_module( 2025-12-04T09:50:51.8566146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8566377Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8566680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8566768Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8567053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8567139Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8567424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8567510Z output = self.layer_1(output) 2025-12-04T09:50:51.8567514Z 2025-12-04T09:50:51.8567626Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8567846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8567927Z res = mod(**inputs) 2025-12-04T09:50:51.8568209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8568307Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8568608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8568703Z outputs = layer_module( 2025-12-04T09:50:51.8568999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8569229Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8569535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8569624Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8569911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8569996Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8570307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8570407Z output = self.activation_function(output) 2025-12-04T09:50:51.8570656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8570733Z return self.act(input) 2025-12-04T09:50:51.8570736Z 2025-12-04T09:50:51.8570853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8571073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8571145Z res = mod(**inputs) 2025-12-04T09:50:51.8571453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8571543Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8571816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8571893Z outputs = layer_module( 2025-12-04T09:50:51.8572153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8572369Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8572634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8572713Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8572979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8573054Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8573317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8573390Z output = self.layer_2(output) 2025-12-04T09:50:51.8573395Z 2025-12-04T09:50:51.8573498Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8573707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8573772Z res = mod(**inputs) 2025-12-04T09:50:51.8574032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8574124Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8574383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8574461Z outputs = layer_module( 2025-12-04T09:50:51.8574720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8574793Z outputs = self.rel_attn( 2025-12-04T09:50:51.8575076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8575195Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8575199Z 2025-12-04T09:50:51.8575310Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8575519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8575589Z res = mod(**inputs) 2025-12-04T09:50:51.8575867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8575958Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8576228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8576305Z outputs = layer_module( 2025-12-04T09:50:51.8576579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8576667Z outputs = self.rel_attn( 2025-12-04T09:50:51.8576942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8577051Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8577055Z 2025-12-04T09:50:51.8577170Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8577381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8577455Z res = mod(**inputs) 2025-12-04T09:50:51.8577731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8577815Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8578080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8578150Z outputs = layer_module( 2025-12-04T09:50:51.8578411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8578513Z outputs = self.rel_attn( 2025-12-04T09:50:51.8578766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8578849Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8579121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8579254Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8579258Z 2025-12-04T09:50:51.8579369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8579566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8579639Z res = mod(**inputs) 2025-12-04T09:50:51.8579898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8579986Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8580252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8580320Z outputs = layer_module( 2025-12-04T09:50:51.8580576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8580655Z outputs = self.rel_attn( 2025-12-04T09:50:51.8580909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8581049Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8581053Z 2025-12-04T09:50:51.8581157Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8581397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8581472Z res = mod(**inputs) 2025-12-04T09:50:51.8581729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8581814Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8582078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8582147Z outputs = layer_module( 2025-12-04T09:50:51.8582411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8582480Z outputs = self.rel_attn( 2025-12-04T09:50:51.8582733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8582841Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8583119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8583260Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8583263Z 2025-12-04T09:50:51.8583365Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8583566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8583640Z res = mod(**inputs) 2025-12-04T09:50:51.8583924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8584009Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8584271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8584342Z outputs = layer_module( 2025-12-04T09:50:51.8584607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8584678Z outputs = self.rel_attn( 2025-12-04T09:50:51.8584948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8585062Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8585066Z 2025-12-04T09:50:51.8585174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8585393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8585462Z res = mod(**inputs) 2025-12-04T09:50:51.8585733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8585831Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8586105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8586179Z outputs = layer_module( 2025-12-04T09:50:51.8586454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8586526Z outputs = self.rel_attn( 2025-12-04T09:50:51.8586814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8586887Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8587162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8587293Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8587297Z 2025-12-04T09:50:51.8587400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8587624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8587707Z res = mod(**inputs) 2025-12-04T09:50:51.8587965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8588054Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8588314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8588383Z outputs = layer_module( 2025-12-04T09:50:51.8588646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8588716Z outputs = self.rel_attn( 2025-12-04T09:50:51.8588975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8589098Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8589377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8589498Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8589502Z 2025-12-04T09:50:51.8589605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8589811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8589877Z res = mod(**inputs) 2025-12-04T09:50:51.8590152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8590243Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8590500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8590571Z outputs = layer_module( 2025-12-04T09:50:51.8590838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8590908Z outputs = self.rel_attn( 2025-12-04T09:50:51.8591168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8591262Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8591539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8591661Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8591664Z 2025-12-04T09:50:51.8591768Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8591977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8592043Z res = mod(**inputs) 2025-12-04T09:50:51.8592303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8592394Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8592651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8592718Z outputs = layer_module( 2025-12-04T09:50:51.8592981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8593195Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8593478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8593564Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8593855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8593959Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8594232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8594310Z output = self.layer_1(output) 2025-12-04T09:50:51.8594321Z 2025-12-04T09:50:51.8594431Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8594643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8594721Z res = mod(**inputs) 2025-12-04T09:50:51.8594996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8595086Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8595383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8595460Z outputs = layer_module( 2025-12-04T09:50:51.8595738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8595959Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8596238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8596328Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8596621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8596700Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8596982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8597080Z output = self.activation_function(output) 2025-12-04T09:50:51.8597320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8597397Z return self.act(input) 2025-12-04T09:50:51.8597400Z 2025-12-04T09:50:51.8597509Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8597724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8597793Z res = mod(**inputs) 2025-12-04T09:50:51.8598069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8598160Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8598432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8598512Z outputs = layer_module( 2025-12-04T09:50:51.8598782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8599004Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8599290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8599375Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8599654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8599732Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8600000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8600084Z output = self.layer_2(output) 2025-12-04T09:50:51.8600087Z 2025-12-04T09:50:51.8600198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8600450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8600591Z res = mod(**inputs) 2025-12-04T09:50:51.8600880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8600978Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8601251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8601326Z outputs = layer_module( 2025-12-04T09:50:51.8601613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8601691Z outputs = self.rel_attn( 2025-12-04T09:50:51.8601972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8602103Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8602109Z 2025-12-04T09:50:51.8602221Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8602440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8602509Z res = mod(**inputs) 2025-12-04T09:50:51.8602791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8602880Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8603171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8603253Z outputs = layer_module( 2025-12-04T09:50:51.8603524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8603600Z outputs = self.rel_attn( 2025-12-04T09:50:51.8603881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8603991Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8603995Z 2025-12-04T09:50:51.8604110Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8604320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8604389Z res = mod(**inputs) 2025-12-04T09:50:51.8604670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8604763Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8605040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8605112Z outputs = layer_module( 2025-12-04T09:50:51.8605385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8605468Z outputs = self.rel_attn( 2025-12-04T09:50:51.8605738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8605818Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8606110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8606245Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8606250Z 2025-12-04T09:50:51.8606359Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8606569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8606637Z res = mod(**inputs) 2025-12-04T09:50:51.8606965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8607083Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8607354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8607435Z outputs = layer_module( 2025-12-04T09:50:51.8607705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8607787Z outputs = self.rel_attn( 2025-12-04T09:50:51.8608058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8608201Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8608204Z 2025-12-04T09:50:51.8608321Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8608550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8608632Z res = mod(**inputs) 2025-12-04T09:50:51.8608913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8609002Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8609291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8609363Z outputs = layer_module( 2025-12-04T09:50:51.8609637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8609739Z outputs = self.rel_attn( 2025-12-04T09:50:51.8610018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8610112Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8610409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8610550Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8610554Z 2025-12-04T09:50:51.8610673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8610886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8610962Z res = mod(**inputs) 2025-12-04T09:50:51.8611241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8611331Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8611624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8611696Z outputs = layer_module( 2025-12-04T09:50:51.8611971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8612053Z outputs = self.rel_attn( 2025-12-04T09:50:51.8612327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8612441Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8612445Z 2025-12-04T09:50:51.8612552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8612765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8612843Z res = mod(**inputs) 2025-12-04T09:50:51.8613118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8613212Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8613519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8613610Z outputs = layer_module( 2025-12-04T09:50:51.8613898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8613972Z outputs = self.rel_attn( 2025-12-04T09:50:51.8614249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8614336Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8614640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8614781Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8614785Z 2025-12-04T09:50:51.8614894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8615127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8615211Z res = mod(**inputs) 2025-12-04T09:50:51.8615489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8615584Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8615874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8615947Z outputs = layer_module( 2025-12-04T09:50:51.8616229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8616320Z outputs = self.rel_attn( 2025-12-04T09:50:51.8616589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8616693Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8616989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8617117Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8617120Z 2025-12-04T09:50:51.8617228Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8617439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8617516Z res = mod(**inputs) 2025-12-04T09:50:51.8617785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8617876Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8618153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8618225Z outputs = layer_module( 2025-12-04T09:50:51.8618504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8618579Z outputs = self.rel_attn( 2025-12-04T09:50:51.8618847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8618949Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8619238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8619363Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8619368Z 2025-12-04T09:50:51.8619475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8619683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8619760Z res = mod(**inputs) 2025-12-04T09:50:51.8620048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8620155Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8620433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8620506Z outputs = layer_module( 2025-12-04T09:50:51.8620780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8621003Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8621286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8621378Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8621667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8621758Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8622042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8622120Z output = self.layer_1(output) 2025-12-04T09:50:51.8622123Z 2025-12-04T09:50:51.8622240Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8622454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8622524Z res = mod(**inputs) 2025-12-04T09:50:51.8622822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8622911Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8623191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8623265Z outputs = layer_module( 2025-12-04T09:50:51.8623548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8623783Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8624072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8624165Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8624449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8624530Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8624817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8624916Z output = self.activation_function(output) 2025-12-04T09:50:51.8625159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8625246Z return self.act(input) 2025-12-04T09:50:51.8625250Z 2025-12-04T09:50:51.8625363Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8625584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8625656Z res = mod(**inputs) 2025-12-04T09:50:51.8625959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8626057Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8626328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8626406Z outputs = layer_module( 2025-12-04T09:50:51.8626688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8626949Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8627245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8627331Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8627614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8627700Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8627979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8628068Z output = self.layer_2(output) 2025-12-04T09:50:51.8628072Z 2025-12-04T09:50:51.8628186Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8628422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8628506Z res = mod(**inputs) 2025-12-04T09:50:51.8628784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8628882Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8629160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8629234Z outputs = layer_module( 2025-12-04T09:50:51.8629519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8629617Z outputs = self.rel_attn( 2025-12-04T09:50:51.8629896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8630016Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8630020Z 2025-12-04T09:50:51.8630136Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8630361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8630430Z res = mod(**inputs) 2025-12-04T09:50:51.8630709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8630809Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8631088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8631172Z outputs = layer_module( 2025-12-04T09:50:51.8631451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8631526Z outputs = self.rel_attn( 2025-12-04T09:50:51.8631815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8631929Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8631933Z 2025-12-04T09:50:51.8632046Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8632269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8632339Z res = mod(**inputs) 2025-12-04T09:50:51.8632626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8632720Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8632999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8633082Z outputs = layer_module( 2025-12-04T09:50:51.8633366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8633490Z outputs = self.rel_attn( 2025-12-04T09:50:51.8633782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8633862Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8634170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8634314Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8634317Z 2025-12-04T09:50:51.8634431Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8634654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8634724Z res = mod(**inputs) 2025-12-04T09:50:51.8635031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8635127Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8635411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8635491Z outputs = layer_module( 2025-12-04T09:50:51.8635771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8635846Z outputs = self.rel_attn( 2025-12-04T09:50:51.8636132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8636314Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8636318Z 2025-12-04T09:50:51.8636439Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8636660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8636736Z res = mod(**inputs) 2025-12-04T09:50:51.8637037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8637134Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8637431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8637509Z outputs = layer_module( 2025-12-04T09:50:51.8637796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8637883Z outputs = self.rel_attn( 2025-12-04T09:50:51.8638171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8638255Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8638575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8638726Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8638730Z 2025-12-04T09:50:51.8638864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8639081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8639153Z res = mod(**inputs) 2025-12-04T09:50:51.8639445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8639541Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8639832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8639909Z outputs = layer_module( 2025-12-04T09:50:51.8640189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8640312Z outputs = self.rel_attn( 2025-12-04T09:50:51.8640664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8640777Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8640790Z 2025-12-04T09:50:51.8640903Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8641115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8641194Z res = mod(**inputs) 2025-12-04T09:50:51.8641477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8641571Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8641884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8641965Z outputs = layer_module( 2025-12-04T09:50:51.8642258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8642335Z outputs = self.rel_attn( 2025-12-04T09:50:51.8642606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8642695Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8642986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8643143Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8643147Z 2025-12-04T09:50:51.8643267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8643485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8643566Z res = mod(**inputs) 2025-12-04T09:50:51.8643841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8643939Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8644219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8644292Z outputs = layer_module( 2025-12-04T09:50:51.8644574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8644648Z outputs = self.rel_attn( 2025-12-04T09:50:51.8644916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8645019Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8645313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8645434Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8645438Z 2025-12-04T09:50:51.8645554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8645765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8645842Z res = mod(**inputs) 2025-12-04T09:50:51.8646115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8646206Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8646486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8646559Z outputs = layer_module( 2025-12-04T09:50:51.8646829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8646946Z outputs = self.rel_attn( 2025-12-04T09:50:51.8647376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8647484Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8647777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8647897Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8647901Z 2025-12-04T09:50:51.8648024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8648236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8648314Z res = mod(**inputs) 2025-12-04T09:50:51.8648585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8648722Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8649009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8649081Z outputs = layer_module( 2025-12-04T09:50:51.8649354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8649588Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8649875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8649999Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8650285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8650363Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8650656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8650738Z output = self.layer_1(output) 2025-12-04T09:50:51.8650742Z 2025-12-04T09:50:51.8650858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8651074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8651144Z res = mod(**inputs) 2025-12-04T09:50:51.8651423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8651523Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8651782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8651858Z outputs = layer_module( 2025-12-04T09:50:51.8652116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8652335Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8672089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8672390Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8672740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8672862Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8673161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8673267Z output = self.activation_function(output) 2025-12-04T09:50:51.8673518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8673804Z return self.act(input) 2025-12-04T09:50:51.8673813Z 2025-12-04T09:50:51.8673950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8674181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8674258Z res = mod(**inputs) 2025-12-04T09:50:51.8674557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8674656Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8674950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8675029Z outputs = layer_module( 2025-12-04T09:50:51.8675309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8675577Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8675871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8675961Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8676246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8676327Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8676613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8676736Z output = self.layer_2(output) 2025-12-04T09:50:51.8676740Z 2025-12-04T09:50:51.8676864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8677107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8677182Z res = mod(**inputs) 2025-12-04T09:50:51.8677489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8677589Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8677877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8677966Z outputs = layer_module( 2025-12-04T09:50:51.8678258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8678339Z outputs = self.rel_attn( 2025-12-04T09:50:51.8678637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8678755Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8678760Z 2025-12-04T09:50:51.8678886Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8679116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8679191Z res = mod(**inputs) 2025-12-04T09:50:51.8679488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8679582Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8679882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8679961Z outputs = layer_module( 2025-12-04T09:50:51.8680250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8680335Z outputs = self.rel_attn( 2025-12-04T09:50:51.8680736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8680919Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8680951Z 2025-12-04T09:50:51.8681069Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8681292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8681374Z res = mod(**inputs) 2025-12-04T09:50:51.8681683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8681778Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8682075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8682151Z outputs = layer_module( 2025-12-04T09:50:51.8682430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8682527Z outputs = self.rel_attn( 2025-12-04T09:50:51.8682807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8682895Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8683191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8683343Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8683355Z 2025-12-04T09:50:51.8683466Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8683697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8683773Z res = mod(**inputs) 2025-12-04T09:50:51.8684047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8684136Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8684417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8684491Z outputs = layer_module( 2025-12-04T09:50:51.8684766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8684839Z outputs = self.rel_attn( 2025-12-04T09:50:51.8685109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8685265Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8685270Z 2025-12-04T09:50:51.8685381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8685591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8685664Z res = mod(**inputs) 2025-12-04T09:50:51.8685941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8686038Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8686309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8686380Z outputs = layer_module( 2025-12-04T09:50:51.8686654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8686727Z outputs = self.rel_attn( 2025-12-04T09:50:51.8687003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8687082Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8687375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8687540Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8687560Z 2025-12-04T09:50:51.8687672Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8687883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8687960Z res = mod(**inputs) 2025-12-04T09:50:51.8688233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8688329Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8688603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8688675Z outputs = layer_module( 2025-12-04T09:50:51.8688953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8689044Z outputs = self.rel_attn( 2025-12-04T09:50:51.8689324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8689436Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8689441Z 2025-12-04T09:50:51.8689559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8689761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8689826Z res = mod(**inputs) 2025-12-04T09:50:51.8690082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8690190Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8690450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8690524Z outputs = layer_module( 2025-12-04T09:50:51.8690781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8690857Z outputs = self.rel_attn( 2025-12-04T09:50:51.8691130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8691208Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8691495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8691639Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8691645Z 2025-12-04T09:50:51.8691754Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8691970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8692040Z res = mod(**inputs) 2025-12-04T09:50:51.8692315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8692414Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8692685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8692765Z outputs = layer_module( 2025-12-04T09:50:51.8693033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8693107Z outputs = self.rel_attn( 2025-12-04T09:50:51.8693382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8693482Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8693775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8693926Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8693946Z 2025-12-04T09:50:51.8694057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8694277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8694347Z res = mod(**inputs) 2025-12-04T09:50:51.8694624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8694723Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8695003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8695082Z outputs = layer_module( 2025-12-04T09:50:51.8695356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8695457Z outputs = self.rel_attn( 2025-12-04T09:50:51.8695736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8695837Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8696128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8696259Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8696263Z 2025-12-04T09:50:51.8696374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8696611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8696680Z res = mod(**inputs) 2025-12-04T09:50:51.8696955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8697051Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8697328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8697409Z outputs = layer_module( 2025-12-04T09:50:51.8697682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8697910Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8698202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8698289Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8698565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8698646Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8698905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8698992Z output = self.layer_1(output) 2025-12-04T09:50:51.8698996Z 2025-12-04T09:50:51.8699108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8699321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8699398Z res = mod(**inputs) 2025-12-04T09:50:51.8699671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8699766Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8700038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8700111Z outputs = layer_module( 2025-12-04T09:50:51.8700389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8700649Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8700936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8701027Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8701301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8701387Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8701662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8701758Z output = self.activation_function(output) 2025-12-04T09:50:51.8702000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8702100Z return self.act(input) 2025-12-04T09:50:51.8702105Z 2025-12-04T09:50:51.8702220Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8702421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8702485Z res = mod(**inputs) 2025-12-04T09:50:51.8702745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8702829Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8703086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8703180Z outputs = layer_module( 2025-12-04T09:50:51.8703437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8703652Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8703924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8704003Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8704283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8704360Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8704641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8704721Z output = self.layer_2(output) 2025-12-04T09:50:51.8704725Z 2025-12-04T09:50:51.8704836Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8705057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8705125Z res = mod(**inputs) 2025-12-04T09:50:51.8705403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8705502Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8705775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8705854Z outputs = layer_module( 2025-12-04T09:50:51.8706123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8706199Z outputs = self.rel_attn( 2025-12-04T09:50:51.8706501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8706606Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8706609Z 2025-12-04T09:50:51.8706721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8706970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8707055Z res = mod(**inputs) 2025-12-04T09:50:51.8707331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8707417Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8707687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8707769Z outputs = layer_module( 2025-12-04T09:50:51.8708050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8708135Z outputs = self.rel_attn( 2025-12-04T09:50:51.8708414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8708546Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8708552Z 2025-12-04T09:50:51.8708672Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8708885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8708955Z res = mod(**inputs) 2025-12-04T09:50:51.8709238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8709327Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8709608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8709701Z outputs = layer_module( 2025-12-04T09:50:51.8709981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8710062Z outputs = self.rel_attn( 2025-12-04T09:50:51.8710353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8710443Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8710743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8710887Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8710891Z 2025-12-04T09:50:51.8711010Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8711225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8711296Z res = mod(**inputs) 2025-12-04T09:50:51.8711587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8711678Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8711967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8712042Z outputs = layer_module( 2025-12-04T09:50:51.8712321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8712405Z outputs = self.rel_attn( 2025-12-04T09:50:51.8712683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8712837Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8712842Z 2025-12-04T09:50:51.8712954Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8713171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8713247Z res = mod(**inputs) 2025-12-04T09:50:51.8713550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8713655Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8713934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8714008Z outputs = layer_module( 2025-12-04T09:50:51.8714287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8714362Z outputs = self.rel_attn( 2025-12-04T09:50:51.8714633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8714722Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8715014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8715184Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8715190Z 2025-12-04T09:50:51.8715306Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8715526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8715607Z res = mod(**inputs) 2025-12-04T09:50:51.8715893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8715984Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8716273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8716368Z outputs = layer_module( 2025-12-04T09:50:51.8716663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8716740Z outputs = self.rel_attn( 2025-12-04T09:50:51.8717027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8717147Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8717151Z 2025-12-04T09:50:51.8717264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8717493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8717562Z res = mod(**inputs) 2025-12-04T09:50:51.8717848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8717946Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8718235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8718308Z outputs = layer_module( 2025-12-04T09:50:51.8718608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8718686Z outputs = self.rel_attn( 2025-12-04T09:50:51.8718979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8719059Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8719368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8719514Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8719519Z 2025-12-04T09:50:51.8719632Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8719859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8719929Z res = mod(**inputs) 2025-12-04T09:50:51.8720222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8720365Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8720752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8720833Z outputs = layer_module( 2025-12-04T09:50:51.8721125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8721203Z outputs = self.rel_attn( 2025-12-04T09:50:51.8721490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8721594Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8721898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8722035Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8722060Z 2025-12-04T09:50:51.8722179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8722400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8722481Z res = mod(**inputs) 2025-12-04T09:50:51.8722786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8722886Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8723193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8723290Z outputs = layer_module( 2025-12-04T09:50:51.8723580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8723658Z outputs = self.rel_attn( 2025-12-04T09:50:51.8723965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8724067Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8724370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8724504Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8724508Z 2025-12-04T09:50:51.8724621Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8724840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8724921Z res = mod(**inputs) 2025-12-04T09:50:51.8725202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8725302Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8725607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8725685Z outputs = layer_module( 2025-12-04T09:50:51.8725973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8726205Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8726507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8726596Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8726880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8726971Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8727275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8727376Z output = self.layer_1(output) 2025-12-04T09:50:51.8727417Z 2025-12-04T09:50:51.8727532Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8727749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8727828Z res = mod(**inputs) 2025-12-04T09:50:51.8728131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8728222Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8728512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8728585Z outputs = layer_module( 2025-12-04T09:50:51.8728867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8729115Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8729405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8729504Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8729783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8729863Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8730147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8730267Z output = self.activation_function(output) 2025-12-04T09:50:51.8730514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8730593Z return self.act(input) 2025-12-04T09:50:51.8730597Z 2025-12-04T09:50:51.8730712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8730940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8731011Z res = mod(**inputs) 2025-12-04T09:50:51.8731316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8731408Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8731688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8731771Z outputs = layer_module( 2025-12-04T09:50:51.8732050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8732277Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8732578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8732666Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8732954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8733032Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8733310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8733398Z output = self.layer_2(output) 2025-12-04T09:50:51.8733404Z 2025-12-04T09:50:51.8733516Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8733738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8733809Z res = mod(**inputs) 2025-12-04T09:50:51.8734088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8734227Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8734507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8734582Z outputs = layer_module( 2025-12-04T09:50:51.8734869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8734946Z outputs = self.rel_attn( 2025-12-04T09:50:51.8735226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8735339Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8735343Z 2025-12-04T09:50:51.8735454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8735700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8735775Z res = mod(**inputs) 2025-12-04T09:50:51.8736063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8736153Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8736433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8736512Z outputs = layer_module( 2025-12-04T09:50:51.8736791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8736886Z outputs = self.rel_attn( 2025-12-04T09:50:51.8737176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8737289Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8737293Z 2025-12-04T09:50:51.8737412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8737633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8737705Z res = mod(**inputs) 2025-12-04T09:50:51.8737997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8738087Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8738373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8738458Z outputs = layer_module( 2025-12-04T09:50:51.8738743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8738826Z outputs = self.rel_attn( 2025-12-04T09:50:51.8739109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8739194Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8739507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8739653Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8739657Z 2025-12-04T09:50:51.8739777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8739993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8740067Z res = mod(**inputs) 2025-12-04T09:50:51.8740360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8740454Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8740749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8740868Z outputs = layer_module( 2025-12-04T09:50:51.8741149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8741236Z outputs = self.rel_attn( 2025-12-04T09:50:51.8741514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8741662Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8741666Z 2025-12-04T09:50:51.8741789Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8742007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8742087Z res = mod(**inputs) 2025-12-04T09:50:51.8742365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8742476Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8742770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8742844Z outputs = layer_module( 2025-12-04T09:50:51.8743122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8743208Z outputs = self.rel_attn( 2025-12-04T09:50:51.8743484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8743591Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8743889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8744031Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8744035Z 2025-12-04T09:50:51.8744158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8744376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8744454Z res = mod(**inputs) 2025-12-04T09:50:51.8744734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8744826Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8745115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8745192Z outputs = layer_module( 2025-12-04T09:50:51.8745472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8745556Z outputs = self.rel_attn( 2025-12-04T09:50:51.8745841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8745962Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8745966Z 2025-12-04T09:50:51.8746079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8746296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8746381Z res = mod(**inputs) 2025-12-04T09:50:51.8746661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8746757Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8747189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8747273Z outputs = layer_module( 2025-12-04T09:50:51.8747565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8747646Z outputs = self.rel_attn( 2025-12-04T09:50:51.8748021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8748112Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8748404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8748546Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8748551Z 2025-12-04T09:50:51.8748663Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8748880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8748959Z res = mod(**inputs) 2025-12-04T09:50:51.8749249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8749374Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8749652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8749725Z outputs = layer_module( 2025-12-04T09:50:51.8750004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8750078Z outputs = self.rel_attn( 2025-12-04T09:50:51.8750345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8750480Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8750771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8750898Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8750902Z 2025-12-04T09:50:51.8751012Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8751226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8751302Z res = mod(**inputs) 2025-12-04T09:50:51.8751574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8751668Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8751941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8752015Z outputs = layer_module( 2025-12-04T09:50:51.8752290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8752364Z outputs = self.rel_attn( 2025-12-04T09:50:51.8752632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8752737Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8753033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8753158Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8753162Z 2025-12-04T09:50:51.8753271Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8753482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8753557Z res = mod(**inputs) 2025-12-04T09:50:51.8753832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8753928Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8754199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8754289Z outputs = layer_module( 2025-12-04T09:50:51.8754603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8754826Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8755109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8755201Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8755476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8755564Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8755838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8755917Z output = self.layer_1(output) 2025-12-04T09:50:51.8755937Z 2025-12-04T09:50:51.8756060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8756271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8756348Z res = mod(**inputs) 2025-12-04T09:50:51.8756623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8756711Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8756991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8757082Z outputs = layer_module( 2025-12-04T09:50:51.8757360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8757593Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8757885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8757978Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8758260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8758337Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8758622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8758718Z output = self.activation_function(output) 2025-12-04T09:50:51.8758964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8759039Z return self.act(input) 2025-12-04T09:50:51.8759043Z 2025-12-04T09:50:51.8759154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8759378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8759448Z res = mod(**inputs) 2025-12-04T09:50:51.8759726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8759823Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8760099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8760177Z outputs = layer_module( 2025-12-04T09:50:51.8760455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8760735Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8761033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8761158Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8761448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8761528Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8761807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8761898Z output = self.layer_2(output) 2025-12-04T09:50:51.8761902Z 2025-12-04T09:50:51.8762015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8762233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8762312Z res = mod(**inputs) 2025-12-04T09:50:51.8762603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8762718Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8762997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8763070Z outputs = layer_module( 2025-12-04T09:50:51.8763349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8763423Z outputs = self.rel_attn( 2025-12-04T09:50:51.8763702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8763829Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8763833Z 2025-12-04T09:50:51.8763943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8764167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8764236Z res = mod(**inputs) 2025-12-04T09:50:51.8764523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8764622Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8764901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8764979Z outputs = layer_module( 2025-12-04T09:50:51.8765279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8765355Z outputs = self.rel_attn( 2025-12-04T09:50:51.8765644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8765751Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8765755Z 2025-12-04T09:50:51.8765871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8766089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8766159Z res = mod(**inputs) 2025-12-04T09:50:51.8766459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8766542Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8766806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8766882Z outputs = layer_module( 2025-12-04T09:50:51.8767146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8767223Z outputs = self.rel_attn( 2025-12-04T09:50:51.8767484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8767560Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8767870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8768018Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8768022Z 2025-12-04T09:50:51.8768132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8768332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8768397Z res = mod(**inputs) 2025-12-04T09:50:51.8768665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8768750Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8769023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8769102Z outputs = layer_module( 2025-12-04T09:50:51.8769394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8769478Z outputs = self.rel_attn( 2025-12-04T09:50:51.8769747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8769888Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8769892Z 2025-12-04T09:50:51.8770007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8770216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8770312Z res = mod(**inputs) 2025-12-04T09:50:51.8770595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8770682Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8770962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8771032Z outputs = layer_module( 2025-12-04T09:50:51.8771287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8771364Z outputs = self.rel_attn( 2025-12-04T09:50:51.8771632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8771718Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8772011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8772151Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8772155Z 2025-12-04T09:50:51.8772271Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8772484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8772556Z res = mod(**inputs) 2025-12-04T09:50:51.8772836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8772923Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8773201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8773275Z outputs = layer_module( 2025-12-04T09:50:51.8773547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8773633Z outputs = self.rel_attn( 2025-12-04T09:50:51.8773904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8774018Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8774056Z 2025-12-04T09:50:51.8774168Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8774381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8774459Z res = mod(**inputs) 2025-12-04T09:50:51.8774742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8774830Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8775121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8775194Z outputs = layer_module( 2025-12-04T09:50:51.8775478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8775553Z outputs = self.rel_attn( 2025-12-04T09:50:51.8775854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8775946Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8776235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8776370Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8776381Z 2025-12-04T09:50:51.8776490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8776706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8776796Z res = mod(**inputs) 2025-12-04T09:50:51.8777060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8777143Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8777412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8777483Z outputs = layer_module( 2025-12-04T09:50:51.8777748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8777819Z outputs = self.rel_attn( 2025-12-04T09:50:51.8778078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8778177Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8778457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8778572Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8778584Z 2025-12-04T09:50:51.8778690Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8778893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8778968Z res = mod(**inputs) 2025-12-04T09:50:51.8779230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8779313Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8779584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8779652Z outputs = layer_module( 2025-12-04T09:50:51.8779916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8779986Z outputs = self.rel_attn( 2025-12-04T09:50:51.8780246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8780345Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8780642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8780772Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8780783Z 2025-12-04T09:50:51.8780887Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8781087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8781159Z res = mod(**inputs) 2025-12-04T09:50:51.8781414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8781499Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8781765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8781833Z outputs = layer_module( 2025-12-04T09:50:51.8782112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8782325Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8782591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8782679Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8782946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8783069Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8783333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8783410Z output = self.layer_1(output) 2025-12-04T09:50:51.8783414Z 2025-12-04T09:50:51.8783532Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8783747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8783815Z res = mod(**inputs) 2025-12-04T09:50:51.8784106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8784194Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8784483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8784557Z outputs = layer_module( 2025-12-04T09:50:51.8784840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8785066Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8785349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8785432Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8785711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8785788Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8786064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8786160Z output = self.activation_function(output) 2025-12-04T09:50:51.8786392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8786475Z return self.act(input) 2025-12-04T09:50:51.8786479Z 2025-12-04T09:50:51.8786589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8786811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8786918Z res = mod(**inputs) 2025-12-04T09:50:51.8787190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8787290Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8787561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8787635Z outputs = layer_module( 2025-12-04T09:50:51.8787910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8788129Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8788414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8788514Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8788794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8788880Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8789152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8789238Z output = self.layer_2(output) 2025-12-04T09:50:51.8789242Z 2025-12-04T09:50:51.8789355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8789569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8789666Z res = mod(**inputs) 2025-12-04T09:50:51.8789939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8790027Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8790308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8790382Z outputs = layer_module( 2025-12-04T09:50:51.8790660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8790736Z outputs = self.rel_attn( 2025-12-04T09:50:51.8791003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8791120Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8791127Z 2025-12-04T09:50:51.8791238Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8791453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8791522Z res = mod(**inputs) 2025-12-04T09:50:51.8791798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8791896Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8792168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8792240Z outputs = layer_module( 2025-12-04T09:50:51.8792520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8792597Z outputs = self.rel_attn( 2025-12-04T09:50:51.8792873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8792984Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8792988Z 2025-12-04T09:50:51.8793100Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8793320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8793424Z res = mod(**inputs) 2025-12-04T09:50:51.8793703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8793799Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8794071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8794151Z outputs = layer_module( 2025-12-04T09:50:51.8794420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8794496Z outputs = self.rel_attn( 2025-12-04T09:50:51.8794776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8794854Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8795178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8795325Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8795329Z 2025-12-04T09:50:51.8795440Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8795664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8795732Z res = mod(**inputs) 2025-12-04T09:50:51.8796011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8796370Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8796650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8796734Z outputs = layer_module( 2025-12-04T09:50:51.8797016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8797097Z outputs = self.rel_attn( 2025-12-04T09:50:51.8797383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8797529Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8797533Z 2025-12-04T09:50:51.8797651Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8797865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8797938Z res = mod(**inputs) 2025-12-04T09:50:51.8798225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8798316Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8798599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8798682Z outputs = layer_module( 2025-12-04T09:50:51.8798966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8799049Z outputs = self.rel_attn( 2025-12-04T09:50:51.8799325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8799406Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8799711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8799855Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8799859Z 2025-12-04T09:50:51.8799979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8800201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8800309Z res = mod(**inputs) 2025-12-04T09:50:51.8800688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8800785Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8801071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8801156Z outputs = layer_module( 2025-12-04T09:50:51.8801439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8801524Z outputs = self.rel_attn( 2025-12-04T09:50:51.8801803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8801915Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8801921Z 2025-12-04T09:50:51.8802068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8802300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8802377Z res = mod(**inputs) 2025-12-04T09:50:51.8802652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8802741Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8803028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8803126Z outputs = layer_module( 2025-12-04T09:50:51.8803406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8803494Z outputs = self.rel_attn( 2025-12-04T09:50:51.8803774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8803868Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8804165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8804302Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8804306Z 2025-12-04T09:50:51.8804429Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8804648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8804729Z res = mod(**inputs) 2025-12-04T09:50:51.8805011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8805101Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8805396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8805474Z outputs = layer_module( 2025-12-04T09:50:51.8805766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8805843Z outputs = self.rel_attn( 2025-12-04T09:50:51.8806129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8806227Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8806529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8806664Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8806667Z 2025-12-04T09:50:51.8806777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8807001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8807091Z res = mod(**inputs) 2025-12-04T09:50:51.8807392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8807489Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8807791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8807864Z outputs = layer_module( 2025-12-04T09:50:51.8808173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8808251Z outputs = self.rel_attn( 2025-12-04T09:50:51.8808539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8808636Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8808960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8809097Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8809101Z 2025-12-04T09:50:51.8809214Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8809441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8809511Z res = mod(**inputs) 2025-12-04T09:50:51.8809813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8809936Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8810219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8810294Z outputs = layer_module( 2025-12-04T09:50:51.8810591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8810822Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8811127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8811215Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8811520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8811608Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8811900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8811989Z output = self.layer_1(output) 2025-12-04T09:50:51.8811992Z 2025-12-04T09:50:51.8812107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8812331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8812410Z res = mod(**inputs) 2025-12-04T09:50:51.8812700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8812790Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8813105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8813179Z outputs = layer_module( 2025-12-04T09:50:51.8813475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8813707Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8814007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8814151Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8814435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8814521Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8814822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8814919Z output = self.activation_function(output) 2025-12-04T09:50:51.8815164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8815242Z return self.act(input) 2025-12-04T09:50:51.8815246Z 2025-12-04T09:50:51.8815358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8815581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8815678Z res = mod(**inputs) 2025-12-04T09:50:51.8815966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8816060Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8816341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8816423Z outputs = layer_module( 2025-12-04T09:50:51.8816701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8816956Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8817252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8817337Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8817630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8817712Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8817989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8818088Z output = self.layer_2(output) 2025-12-04T09:50:51.8818091Z 2025-12-04T09:50:51.8818202Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8818418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8818489Z res = mod(**inputs) 2025-12-04T09:50:51.8818760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8818860Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8819140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8819219Z outputs = layer_module( 2025-12-04T09:50:51.8819507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8819583Z outputs = self.rel_attn( 2025-12-04T09:50:51.8819868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8819979Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8819983Z 2025-12-04T09:50:51.8820099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8820327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8820399Z res = mod(**inputs) 2025-12-04T09:50:51.8820694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8820811Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8821126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8821207Z outputs = layer_module( 2025-12-04T09:50:51.8821480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8821556Z outputs = self.rel_attn( 2025-12-04T09:50:51.8821832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8821945Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8821948Z 2025-12-04T09:50:51.8822065Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8822283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8822373Z res = mod(**inputs) 2025-12-04T09:50:51.8822667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8822759Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8823050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8823124Z outputs = layer_module( 2025-12-04T09:50:51.8823402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8823505Z outputs = self.rel_attn( 2025-12-04T09:50:51.8823785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8823867Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8824180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8824330Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8824334Z 2025-12-04T09:50:51.8824453Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8824671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8824743Z res = mod(**inputs) 2025-12-04T09:50:51.8825038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8825129Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8825419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8825493Z outputs = layer_module( 2025-12-04T09:50:51.8825776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8825862Z outputs = self.rel_attn( 2025-12-04T09:50:51.8826144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8826293Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8826297Z 2025-12-04T09:50:51.8826417Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8826634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8826714Z res = mod(**inputs) 2025-12-04T09:50:51.8826997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8827087Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8827378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8827458Z outputs = layer_module( 2025-12-04T09:50:51.8827778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8827865Z outputs = self.rel_attn( 2025-12-04T09:50:51.8828142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8828230Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8828527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8828671Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8828675Z 2025-12-04T09:50:51.8828796Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8829013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8829106Z res = mod(**inputs) 2025-12-04T09:50:51.8829392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8829486Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8829773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8829848Z outputs = layer_module( 2025-12-04T09:50:51.8830126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8830228Z outputs = self.rel_attn( 2025-12-04T09:50:51.8830520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8830638Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8830642Z 2025-12-04T09:50:51.8830756Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8830980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8831063Z res = mod(**inputs) 2025-12-04T09:50:51.8831356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8831452Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8831744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8831819Z outputs = layer_module( 2025-12-04T09:50:51.8832120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8832196Z outputs = self.rel_attn( 2025-12-04T09:50:51.8832484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8832573Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8832890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8833034Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8833037Z 2025-12-04T09:50:51.8833150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8833374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8833452Z res = mod(**inputs) 2025-12-04T09:50:51.8833747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8833848Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8834152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8834227Z outputs = layer_module( 2025-12-04T09:50:51.8834547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8834640Z outputs = self.rel_attn( 2025-12-04T09:50:51.8834931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8835039Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8835352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8835487Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8835491Z 2025-12-04T09:50:51.8835601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8835824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8835904Z res = mod(**inputs) 2025-12-04T09:50:51.8836214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8836316Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8836603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8836678Z outputs = layer_module( 2025-12-04T09:50:51.8836965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8837043Z outputs = self.rel_attn( 2025-12-04T09:50:51.8837343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8837449Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8837759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8837892Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8837896Z 2025-12-04T09:50:51.8838009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8838229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8838309Z res = mod(**inputs) 2025-12-04T09:50:51.8838591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8838693Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8838978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8839052Z outputs = layer_module( 2025-12-04T09:50:51.8839339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8839572Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8839866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8839961Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8840243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8840328Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8840693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8840786Z output = self.layer_1(output) 2025-12-04T09:50:51.8840790Z 2025-12-04T09:50:51.8840917Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8841135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8841244Z res = mod(**inputs) 2025-12-04T09:50:51.8841549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8841640Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8841933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8842009Z outputs = layer_module( 2025-12-04T09:50:51.8842288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8842527Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8842817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8842935Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8843222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8843304Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8843590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8843689Z output = self.activation_function(output) 2025-12-04T09:50:51.8843937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8844037Z return self.act(input) 2025-12-04T09:50:51.8844041Z 2025-12-04T09:50:51.8844156Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8844380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8844451Z res = mod(**inputs) 2025-12-04T09:50:51.8844734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8844835Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8845113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8845195Z outputs = layer_module( 2025-12-04T09:50:51.8845472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8845697Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8845995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8846080Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8846370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8846454Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8846730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8846817Z output = self.layer_2(output) 2025-12-04T09:50:51.8846820Z 2025-12-04T09:50:51.8846933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8847377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8847464Z res = mod(**inputs) 2025-12-04T09:50:51.8847756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8847856Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8848141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8848218Z outputs = layer_module( 2025-12-04T09:50:51.8848598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8848675Z outputs = self.rel_attn( 2025-12-04T09:50:51.8848959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8849076Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8849084Z 2025-12-04T09:50:51.8849196Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8849421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8849492Z res = mod(**inputs) 2025-12-04T09:50:51.8849773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8849874Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8850184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8850270Z outputs = layer_module( 2025-12-04T09:50:51.8850554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8850629Z outputs = self.rel_attn( 2025-12-04T09:50:51.8850906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8851014Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8851047Z 2025-12-04T09:50:51.8851156Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8851373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8851442Z res = mod(**inputs) 2025-12-04T09:50:51.8851724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8851815Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8852084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8852164Z outputs = layer_module( 2025-12-04T09:50:51.8852431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8852513Z outputs = self.rel_attn( 2025-12-04T09:50:51.8852780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8852861Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8853156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8853303Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8853309Z 2025-12-04T09:50:51.8853418Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8853635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8853703Z res = mod(**inputs) 2025-12-04T09:50:51.8853982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8854069Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8854339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8854421Z outputs = layer_module( 2025-12-04T09:50:51.8854691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8854772Z outputs = self.rel_attn( 2025-12-04T09:50:51.8855062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8855224Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8855227Z 2025-12-04T09:50:51.8855344Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8855557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8855624Z res = mod(**inputs) 2025-12-04T09:50:51.8855907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8855997Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8856277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8856349Z outputs = layer_module( 2025-12-04T09:50:51.8856642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8856727Z outputs = self.rel_attn( 2025-12-04T09:50:51.8856997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8857074Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8857374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8857513Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8857535Z 2025-12-04T09:50:51.8857653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8857874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8857943Z res = mod(**inputs) 2025-12-04T09:50:51.8858224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8858314Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8858591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8858663Z outputs = layer_module( 2025-12-04T09:50:51.8858936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8859019Z outputs = self.rel_attn( 2025-12-04T09:50:51.8859298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8859406Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8859417Z 2025-12-04T09:50:51.8859525Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8859750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8859830Z res = mod(**inputs) 2025-12-04T09:50:51.8860102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8860191Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8860473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8860547Z outputs = layer_module( 2025-12-04T09:50:51.8860835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8860911Z outputs = self.rel_attn( 2025-12-04T09:50:51.8861193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8861278Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8861588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8861740Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8861753Z 2025-12-04T09:50:51.8861862Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8862085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8862162Z res = mod(**inputs) 2025-12-04T09:50:51.8862434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8862524Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8862802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8862875Z outputs = layer_module( 2025-12-04T09:50:51.8863190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8863271Z outputs = self.rel_attn( 2025-12-04T09:50:51.8863552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8863653Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8863945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8864065Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8864094Z 2025-12-04T09:50:51.8864203Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8864426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8864501Z res = mod(**inputs) 2025-12-04T09:50:51.8864776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8864867Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8865144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8865216Z outputs = layer_module( 2025-12-04T09:50:51.8865504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8865578Z outputs = self.rel_attn( 2025-12-04T09:50:51.8865858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8865958Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8866284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8866410Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8866417Z 2025-12-04T09:50:51.8866535Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8866764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8866842Z res = mod(**inputs) 2025-12-04T09:50:51.8867126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8867215Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8867494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8867570Z outputs = layer_module( 2025-12-04T09:50:51.8867859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8868096Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8868428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8868523Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8868806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8868885Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8869171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8869253Z output = self.layer_1(output) 2025-12-04T09:50:51.8869257Z 2025-12-04T09:50:51.8869377Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8869592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8869663Z res = mod(**inputs) 2025-12-04T09:50:51.8869969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8870066Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8870354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8870429Z outputs = layer_module( 2025-12-04T09:50:51.8870706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8870939Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8871259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8871346Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8871637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8871719Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8872005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8872102Z output = self.activation_function(output) 2025-12-04T09:50:51.8872338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8872427Z return self.act(input) 2025-12-04T09:50:51.8872431Z 2025-12-04T09:50:51.8872547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8872770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8872842Z res = mod(**inputs) 2025-12-04T09:50:51.8873123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8873224Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8873506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8873588Z outputs = layer_module( 2025-12-04T09:50:51.8873874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8874096Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8874392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8874478Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8874759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8874849Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8875168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8875257Z output = self.layer_2(output) 2025-12-04T09:50:51.8875261Z 2025-12-04T09:50:51.8875374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8875591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8875670Z res = mod(**inputs) 2025-12-04T09:50:51.8875951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8876044Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8876328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8876402Z outputs = layer_module( 2025-12-04T09:50:51.8876725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8876805Z outputs = self.rel_attn( 2025-12-04T09:50:51.8877087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:50:51.8877207Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:50:51.8877211Z 2025-12-04T09:50:51.8877325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8877550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8877641Z res = mod(**inputs) 2025-12-04T09:50:51.8877925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8878021Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8878306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8878381Z outputs = layer_module( 2025-12-04T09:50:51.8878666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8878743Z outputs = self.rel_attn( 2025-12-04T09:50:51.8879029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:50:51.8879139Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:50:51.8879143Z 2025-12-04T09:50:51.8879257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8879481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8879551Z res = mod(**inputs) 2025-12-04T09:50:51.8879833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8879932Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8880213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8880292Z outputs = layer_module( 2025-12-04T09:50:51.8880640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8880723Z outputs = self.rel_attn( 2025-12-04T09:50:51.8881017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8881103Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8881411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:50:51.8881557Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:50:51.8881562Z 2025-12-04T09:50:51.8881697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8881941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8882014Z res = mod(**inputs) 2025-12-04T09:50:51.8882296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8882395Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8882676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8882758Z outputs = layer_module( 2025-12-04T09:50:51.8883034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8883111Z outputs = self.rel_attn( 2025-12-04T09:50:51.8883414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:50:51.8883567Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:50:51.8883571Z 2025-12-04T09:50:51.8883693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8883910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8883982Z res = mod(**inputs) 2025-12-04T09:50:51.8884274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8884384Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8884671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8884750Z outputs = layer_module( 2025-12-04T09:50:51.8885022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8885106Z outputs = self.rel_attn( 2025-12-04T09:50:51.8885376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8885453Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8885751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:50:51.8885888Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:50:51.8885892Z 2025-12-04T09:50:51.8886011Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8886222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8886290Z res = mod(**inputs) 2025-12-04T09:50:51.8886571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8886661Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8886934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8887014Z outputs = layer_module( 2025-12-04T09:50:51.8887280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8887360Z outputs = self.rel_attn( 2025-12-04T09:50:51.8887627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:50:51.8887737Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:50:51.8887740Z 2025-12-04T09:50:51.8887857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8888067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8888137Z res = mod(**inputs) 2025-12-04T09:50:51.8888461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8888553Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8888833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8888905Z outputs = layer_module( 2025-12-04T09:50:51.8889178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8889263Z outputs = self.rel_attn( 2025-12-04T09:50:51.8889547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:50:51.8889630Z attn_vec = self.rel_attn_core( 2025-12-04T09:50:51.8889989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:50:51.8890128Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:50:51.8890132Z 2025-12-04T09:50:51.8890249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8890460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8890529Z res = mod(**inputs) 2025-12-04T09:50:51.8890813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8890904Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8891209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8891281Z outputs = layer_module( 2025-12-04T09:50:51.8891557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8891638Z outputs = self.rel_attn( 2025-12-04T09:50:51.8891912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8892013Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8892309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8892429Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8892433Z 2025-12-04T09:50:51.8892549Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8892761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8892829Z res = mod(**inputs) 2025-12-04T09:50:51.8893112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8893202Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8893485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8893556Z outputs = layer_module( 2025-12-04T09:50:51.8893827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:50:51.8893910Z outputs = self.rel_attn( 2025-12-04T09:50:51.8894181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:50:51.8894285Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:50:51.8894581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:50:51.8894701Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:50:51.8894705Z 2025-12-04T09:50:51.8894842Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8895074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8895142Z res = mod(**inputs) 2025-12-04T09:50:51.8895425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8895513Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8895792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8895865Z outputs = layer_module( 2025-12-04T09:50:51.8896136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8896365Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8896679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8896777Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8897057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8897137Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8897417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:50:51.8897497Z output = self.layer_1(output) 2025-12-04T09:50:51.8897519Z 2025-12-04T09:50:51.8897631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8897850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8897918Z res = mod(**inputs) 2025-12-04T09:50:51.8898197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8898287Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8898557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8898636Z outputs = layer_module( 2025-12-04T09:50:51.8898906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8899133Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8899418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8899500Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8899781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8899859Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8900147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:50:51.8900247Z output = self.activation_function(output) 2025-12-04T09:50:51.8900484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:50:51.8900568Z return self.act(input) 2025-12-04T09:50:51.8900572Z 2025-12-04T09:50:51.8900681Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8900893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8900970Z res = mod(**inputs) 2025-12-04T09:50:51.8901240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:50:51.8901337Z transformer_outputs = self.transformer( 2025-12-04T09:50:51.8901628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:50:51.8901719Z outputs = layer_module( 2025-12-04T09:50:51.8902008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:50:51.8902226Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:50:51.8902508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:50:51.8902599Z return forward_fn(*input_tensors) 2025-12-04T09:50:51.8902874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:50:51.8902959Z output_x = self.ff(output_x) 2025-12-04T09:50:51.8903263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:50:51.8903344Z output = self.layer_2(output) 2025-12-04T09:50:51.8903348Z 2025-12-04T09:50:51.8903468Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8903692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8903771Z res = mod(**inputs) 2025-12-04T09:50:51.8904045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1633, in forward 2025-12-04T09:50:51.8904166Z logits = self.lm_loss(transformer_outputs[0]) 2025-12-04T09:50:51.8904170Z 2025-12-04T09:50:51.8904285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:50:51.8904508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:50:51.8904576Z res = mod(**inputs) 2025-12-04T09:50:51.8904864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1639, in forward 2025-12-04T09:50:51.8905010Z loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1)) 2025-12-04T09:50:51.8905014Z 2025-12-04T09:51:06.5668507Z Compilation time (from dynamo_timed): 35.491489534 2025-12-04T09:51:06.5715195Z pass 2025-12-04T09:51:06.5715582Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:51:06.5716445Z TIMING: _recursive_pre_grad_passes:0.0141 _recursive_joint_graph_passes:1.41578 _recursive_post_grad_passes:0.59555 async_compile.wait:1.03855 code_gen:13.649 inductor_compile:19.04835 backend_compile:28.82872 gc:0.00039 entire_frame_compile:35.49149 total_wall_time:35.49149 2025-12-04T09:51:06.5717471Z STATS: call_* op count: 818 | FakeTensorMode.__torch_dispatch__:32908 | FakeTensor.__torch_dispatch__:15989 | ProxyTorchDispatchMode.__torch_dispatch__:6774 2025-12-04T09:51:06.5718027Z Dynamo produced 1 graphs covering 818 ops with 0 graph breaks (0 unique) 2025-12-04T09:51:10.0837970Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:51:10.0838938Z import pynvml # type: ignore[import] 2025-12-04T09:51:13.5634606Z 2025-12-04T09:51:14.6696812Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:51:14.6697229Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:51:14.6712956Z cpu eval YituTechConvBert 2025-12-04T09:51:15.5658103Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:51:15.8645754Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:51:16.2122669Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:51:28.7302945Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7304490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7306556Z res = mod(**inputs) 2025-12-04T09:51:28.7307096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7307609Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7308110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7308631Z hidden_states = self.encoder( 2025-12-04T09:51:28.7309088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7309551Z layer_outputs = layer_module( 2025-12-04T09:51:28.7310303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7310818Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7311291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7311752Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7312208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7312659Z self_outputs = self.self( 2025-12-04T09:51:28.7313160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.7313638Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.7313805Z 2025-12-04T09:51:28.7313928Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7314376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7314736Z res = mod(**inputs) 2025-12-04T09:51:28.7315160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7315633Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7316100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7316551Z hidden_states = self.encoder( 2025-12-04T09:51:28.7316973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7317421Z layer_outputs = layer_module( 2025-12-04T09:51:28.7317813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7318238Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7318702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7319161Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7319609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7320055Z self_outputs = self.self( 2025-12-04T09:51:28.7320505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.7321351Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.7321514Z 2025-12-04T09:51:28.7321642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7322062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7322453Z res = mod(**inputs) 2025-12-04T09:51:28.7322944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7323474Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7323942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7324400Z hidden_states = self.encoder( 2025-12-04T09:51:28.7324854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7325315Z layer_outputs = layer_module( 2025-12-04T09:51:28.7325706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7326127Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7326629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7327131Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7327617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7328075Z self_outputs = self.self( 2025-12-04T09:51:28.7328517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.7328994Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.7329168Z 2025-12-04T09:51:28.7329298Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7329550Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7329810Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7330232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7330596Z res = mod(**inputs) 2025-12-04T09:51:28.7331028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7331490Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7331961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7332420Z hidden_states = self.encoder( 2025-12-04T09:51:28.7332861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7333325Z layer_outputs = layer_module( 2025-12-04T09:51:28.7333739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7334154Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7334599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7335065Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7335517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7335955Z self_outputs = self.self( 2025-12-04T09:51:28.7336385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.7336863Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.7337034Z 2025-12-04T09:51:28.7337132Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7337395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7337818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7338176Z res = mod(**inputs) 2025-12-04T09:51:28.7338670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7339144Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7339600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7340046Z hidden_states = self.encoder( 2025-12-04T09:51:28.7340492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7341004Z layer_outputs = layer_module( 2025-12-04T09:51:28.7341395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7341909Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7342411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7342913Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7343371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7343822Z self_outputs = self.self( 2025-12-04T09:51:28.7344245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7344788Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7345338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.7345812Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.7345972Z 2025-12-04T09:51:28.7346091Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7346503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7346871Z res = mod(**inputs) 2025-12-04T09:51:28.7347487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7347957Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7348414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7348876Z hidden_states = self.encoder( 2025-12-04T09:51:28.7349309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7349756Z layer_outputs = layer_module( 2025-12-04T09:51:28.7350154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7350543Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7350986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7351432Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7351879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7352319Z self_outputs = self.self( 2025-12-04T09:51:28.7352750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7353297Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7353826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.7354250Z x = self.pointwise(x) 2025-12-04T09:51:28.7354376Z 2025-12-04T09:51:28.7354489Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7354946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7355314Z res = mod(**inputs) 2025-12-04T09:51:28.7355739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7356182Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7356625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7357071Z hidden_states = self.encoder( 2025-12-04T09:51:28.7357519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7357981Z layer_outputs = layer_module( 2025-12-04T09:51:28.7358360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7358806Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7359262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7359728Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7360193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7360717Z self_outputs = self.self( 2025-12-04T09:51:28.7361158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.7361794Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.7362022Z 2025-12-04T09:51:28.7362136Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7362523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7362874Z res = mod(**inputs) 2025-12-04T09:51:28.7363290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7363749Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7364212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7364672Z hidden_states = self.encoder( 2025-12-04T09:51:28.7365107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7365565Z layer_outputs = layer_module( 2025-12-04T09:51:28.7365949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7366354Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7366801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7367258Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7367712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7368145Z self_outputs = self.self( 2025-12-04T09:51:28.7368571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.7369065Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.7369262Z 2025-12-04T09:51:28.7369389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7369781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7370141Z res = mod(**inputs) 2025-12-04T09:51:28.7370597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7371113Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7371557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7372000Z hidden_states = self.encoder( 2025-12-04T09:51:28.7372434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7372870Z layer_outputs = layer_module( 2025-12-04T09:51:28.7373255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7373655Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7374112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7374587Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7375040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7375496Z self_outputs = self.self( 2025-12-04T09:51:28.7375904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.7376413Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.7376613Z 2025-12-04T09:51:28.7376703Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7376966Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7377215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7377602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7377952Z res = mod(**inputs) 2025-12-04T09:51:28.7378367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7378804Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7379262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7379717Z hidden_states = self.encoder( 2025-12-04T09:51:28.7380156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7380620Z layer_outputs = layer_module( 2025-12-04T09:51:28.7381008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7381414Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7381868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7382330Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7382774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7383206Z self_outputs = self.self( 2025-12-04T09:51:28.7383635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.7384169Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.7384347Z 2025-12-04T09:51:28.7384466Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7384848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7385195Z res = mod(**inputs) 2025-12-04T09:51:28.7385605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7386061Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7386528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7386980Z hidden_states = self.encoder( 2025-12-04T09:51:28.7387405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7387826Z layer_outputs = layer_module( 2025-12-04T09:51:28.7388198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7388594Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7389027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7389462Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7389933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.7390451Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.7390957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.7391414Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7391581Z 2025-12-04T09:51:28.7391696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7392101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7392480Z res = mod(**inputs) 2025-12-04T09:51:28.7392895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7393352Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7393812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7394252Z hidden_states = self.encoder( 2025-12-04T09:51:28.7394697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7395144Z layer_outputs = layer_module( 2025-12-04T09:51:28.7395533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7395936Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7396384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7396852Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7397298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7397748Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7398237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7398783Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7399274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.7399730Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7399893Z 2025-12-04T09:51:28.7400011Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7400416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7400879Z res = mod(**inputs) 2025-12-04T09:51:28.7401302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7401765Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7402283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7402734Z hidden_states = self.encoder( 2025-12-04T09:51:28.7403173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7403620Z layer_outputs = layer_module( 2025-12-04T09:51:28.7403999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7404409Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7404862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7405320Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7405792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7406760Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7407260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7407810Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7408321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.7408826Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.7409285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.7409672Z return self.act(input) 2025-12-04T09:51:28.7409811Z 2025-12-04T09:51:28.7409928Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7410341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7410711Z res = mod(**inputs) 2025-12-04T09:51:28.7411126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7411590Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7412050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7412502Z hidden_states = self.encoder( 2025-12-04T09:51:28.7412945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7413402Z layer_outputs = layer_module( 2025-12-04T09:51:28.7413797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7414201Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7414666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7415137Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7415598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7416037Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7416524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.7417076Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.7417583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.7418047Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7418212Z 2025-12-04T09:51:28.7418358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7418778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7419147Z res = mod(**inputs) 2025-12-04T09:51:28.7419573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7420091Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7420547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7420981Z hidden_states = self.encoder( 2025-12-04T09:51:28.7421416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7421866Z layer_outputs = layer_module( 2025-12-04T09:51:28.7422287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7422694Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7423149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7423602Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7424050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7424532Z self_outputs = self.self( 2025-12-04T09:51:28.7425026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.7425517Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.7425681Z 2025-12-04T09:51:28.7425794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7426184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7426539Z res = mod(**inputs) 2025-12-04T09:51:28.7426939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7427385Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7427826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7428255Z hidden_states = self.encoder( 2025-12-04T09:51:28.7428684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7429120Z layer_outputs = layer_module( 2025-12-04T09:51:28.7429499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7429887Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7430329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7430774Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7431224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7431650Z self_outputs = self.self( 2025-12-04T09:51:28.7432071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.7432521Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.7432674Z 2025-12-04T09:51:28.7432795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7433180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7433537Z res = mod(**inputs) 2025-12-04T09:51:28.7433968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7434428Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7434878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7435316Z hidden_states = self.encoder( 2025-12-04T09:51:28.7435754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7436182Z layer_outputs = layer_module( 2025-12-04T09:51:28.7436564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7436961Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7437401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7437862Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7438304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7438746Z self_outputs = self.self( 2025-12-04T09:51:28.7439163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.7439634Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.7439808Z 2025-12-04T09:51:28.7439899Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7440166Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7440432Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7440928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7441298Z res = mod(**inputs) 2025-12-04T09:51:28.7441717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7442185Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7442661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7443200Z hidden_states = self.encoder( 2025-12-04T09:51:28.7443645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7444096Z layer_outputs = layer_module( 2025-12-04T09:51:28.7444492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7444902Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7445356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7445821Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7446280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7446734Z self_outputs = self.self( 2025-12-04T09:51:28.7447359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.7447852Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.7448029Z 2025-12-04T09:51:28.7448129Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7448395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7448798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7449157Z res = mod(**inputs) 2025-12-04T09:51:28.7449570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7450127Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7450590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7451040Z hidden_states = self.encoder( 2025-12-04T09:51:28.7451474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7451923Z layer_outputs = layer_module( 2025-12-04T09:51:28.7452311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7452726Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7453158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7453609Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7454082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7454515Z self_outputs = self.self( 2025-12-04T09:51:28.7454936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7455471Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7456001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.7456471Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.7456622Z 2025-12-04T09:51:28.7456736Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7457127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7457495Z res = mod(**inputs) 2025-12-04T09:51:28.7457897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7458343Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7458784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7459219Z hidden_states = self.encoder( 2025-12-04T09:51:28.7459656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7460089Z layer_outputs = layer_module( 2025-12-04T09:51:28.7460461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7460856Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7461308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7461753Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7462189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7462630Z self_outputs = self.self( 2025-12-04T09:51:28.7463061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7463596Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7464130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.7464572Z x = self.pointwise(x) 2025-12-04T09:51:28.7464697Z 2025-12-04T09:51:28.7464807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7465197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7465581Z res = mod(**inputs) 2025-12-04T09:51:28.7465988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7466423Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7466877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7467318Z hidden_states = self.encoder( 2025-12-04T09:51:28.7467743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7468154Z layer_outputs = layer_module( 2025-12-04T09:51:28.7468502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7468871Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7469308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7469752Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7470178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7470593Z self_outputs = self.self( 2025-12-04T09:51:28.7471014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.7471531Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.7471749Z 2025-12-04T09:51:28.7471854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7472217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7472562Z res = mod(**inputs) 2025-12-04T09:51:28.7472958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7473404Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7473848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7474293Z hidden_states = self.encoder( 2025-12-04T09:51:28.7474685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7475094Z layer_outputs = layer_module( 2025-12-04T09:51:28.7475444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7475807Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7476219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7476645Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7477079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7477496Z self_outputs = self.self( 2025-12-04T09:51:28.7477911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.7478392Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.7478578Z 2025-12-04T09:51:28.7478698Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7479076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7479421Z res = mod(**inputs) 2025-12-04T09:51:28.7479826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7480289Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7480857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7481317Z hidden_states = self.encoder( 2025-12-04T09:51:28.7481758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7482199Z layer_outputs = layer_module( 2025-12-04T09:51:28.7482551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7482926Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7483329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7483747Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7484197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7484634Z self_outputs = self.self( 2025-12-04T09:51:28.7485045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.7485541Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.7485743Z 2025-12-04T09:51:28.7485834Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7486075Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7486332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7486706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7487060Z res = mod(**inputs) 2025-12-04T09:51:28.7487467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7487917Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7488366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7488805Z hidden_states = self.encoder( 2025-12-04T09:51:28.7489231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7489674Z layer_outputs = layer_module( 2025-12-04T09:51:28.7490054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7490446Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7490892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7491339Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7491788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7492220Z self_outputs = self.self( 2025-12-04T09:51:28.7492643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.7493127Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.7493311Z 2025-12-04T09:51:28.7493434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7493820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7494178Z res = mod(**inputs) 2025-12-04T09:51:28.7494588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7495033Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7495505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7495963Z hidden_states = self.encoder( 2025-12-04T09:51:28.7496386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7496807Z layer_outputs = layer_module( 2025-12-04T09:51:28.7497179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7497572Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7497975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7498390Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7498803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.7499293Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.7499784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.7500246Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7500404Z 2025-12-04T09:51:28.7500517Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7500906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7501249Z res = mod(**inputs) 2025-12-04T09:51:28.7501673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7502114Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7502548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7502979Z hidden_states = self.encoder( 2025-12-04T09:51:28.7503401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7503831Z layer_outputs = layer_module( 2025-12-04T09:51:28.7504193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7504584Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7505022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7505467Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7505895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7506322Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7506786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7507297Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7507776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.7508220Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7508373Z 2025-12-04T09:51:28.7508494Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7508874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7509222Z res = mod(**inputs) 2025-12-04T09:51:28.7509641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7510093Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7510560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7511012Z hidden_states = self.encoder( 2025-12-04T09:51:28.7511439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7511872Z layer_outputs = layer_module( 2025-12-04T09:51:28.7512256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7512650Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7513142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7513586Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7514044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7514487Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7514957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7515470Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7515954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.7516441Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.7516854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.7517277Z return self.act(input) 2025-12-04T09:51:28.7517410Z 2025-12-04T09:51:28.7517528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7517938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7518303Z res = mod(**inputs) 2025-12-04T09:51:28.7518724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7519197Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7519665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7520120Z hidden_states = self.encoder( 2025-12-04T09:51:28.7520641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7521108Z layer_outputs = layer_module( 2025-12-04T09:51:28.7521500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7521907Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7522390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7522864Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7523299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7523737Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7524216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.7524749Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.7525244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.7525688Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7525837Z 2025-12-04T09:51:28.7525961Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7526368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7526745Z res = mod(**inputs) 2025-12-04T09:51:28.7527187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7527637Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7528080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7528509Z hidden_states = self.encoder( 2025-12-04T09:51:28.7528936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7529377Z layer_outputs = layer_module( 2025-12-04T09:51:28.7529745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7530159Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7530602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7531067Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7531517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7531964Z self_outputs = self.self( 2025-12-04T09:51:28.7532393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.7532861Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.7533029Z 2025-12-04T09:51:28.7533140Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7533527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7533870Z res = mod(**inputs) 2025-12-04T09:51:28.7534275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7534717Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7535164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7535601Z hidden_states = self.encoder( 2025-12-04T09:51:28.7536035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7536463Z layer_outputs = layer_module( 2025-12-04T09:51:28.7536837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7537218Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7537652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7538090Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7538527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7538976Z self_outputs = self.self( 2025-12-04T09:51:28.7539388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.7539831Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.7539979Z 2025-12-04T09:51:28.7540091Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7540487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7540852Z res = mod(**inputs) 2025-12-04T09:51:28.7541246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7541762Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7542235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7542691Z hidden_states = self.encoder( 2025-12-04T09:51:28.7543116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7543557Z layer_outputs = layer_module( 2025-12-04T09:51:28.7543943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7544347Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7544794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7545348Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7545805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7546241Z self_outputs = self.self( 2025-12-04T09:51:28.7546677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.7547326Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.7547498Z 2025-12-04T09:51:28.7547597Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7547833Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7548165Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7548568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7548923Z res = mod(**inputs) 2025-12-04T09:51:28.7549341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7549802Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7550259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7550708Z hidden_states = self.encoder( 2025-12-04T09:51:28.7551143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7551590Z layer_outputs = layer_module( 2025-12-04T09:51:28.7551983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7552379Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7552826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7553282Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7553734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7554157Z self_outputs = self.self( 2025-12-04T09:51:28.7554557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.7555021Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.7555192Z 2025-12-04T09:51:28.7555279Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7555537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7555927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7556267Z res = mod(**inputs) 2025-12-04T09:51:28.7556667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7557115Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7557638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7558076Z hidden_states = self.encoder( 2025-12-04T09:51:28.7558514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7558959Z layer_outputs = layer_module( 2025-12-04T09:51:28.7559347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7559745Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7560193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7560714Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7561212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7561671Z self_outputs = self.self( 2025-12-04T09:51:28.7562101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7562662Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7563205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.7563663Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.7563834Z 2025-12-04T09:51:28.7563947Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7564334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7564673Z res = mod(**inputs) 2025-12-04T09:51:28.7565082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7565526Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7565955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7566397Z hidden_states = self.encoder( 2025-12-04T09:51:28.7566818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7567261Z layer_outputs = layer_module( 2025-12-04T09:51:28.7567627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7568018Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7568455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7568907Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7569339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7569767Z self_outputs = self.self( 2025-12-04T09:51:28.7570180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7570706Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7571229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.7571661Z x = self.pointwise(x) 2025-12-04T09:51:28.7571781Z 2025-12-04T09:51:28.7571901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7572282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7572630Z res = mod(**inputs) 2025-12-04T09:51:28.7573088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7573531Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7573963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7574389Z hidden_states = self.encoder( 2025-12-04T09:51:28.7574811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7575237Z layer_outputs = layer_module( 2025-12-04T09:51:28.7575612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7576001Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7576458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7576892Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7577330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7577760Z self_outputs = self.self( 2025-12-04T09:51:28.7578173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.7578688Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.7578943Z 2025-12-04T09:51:28.7579054Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7579444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7579784Z res = mod(**inputs) 2025-12-04T09:51:28.7580193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7580632Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7581071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7581492Z hidden_states = self.encoder( 2025-12-04T09:51:28.7581917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7582343Z layer_outputs = layer_module( 2025-12-04T09:51:28.7582709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7583100Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7583531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7583985Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7584419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7584847Z self_outputs = self.self( 2025-12-04T09:51:28.7585259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.7585752Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.7585938Z 2025-12-04T09:51:28.7586049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7586435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7586781Z res = mod(**inputs) 2025-12-04T09:51:28.7587177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7587622Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7588117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7588552Z hidden_states = self.encoder( 2025-12-04T09:51:28.7588964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7589396Z layer_outputs = layer_module( 2025-12-04T09:51:28.7589771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7590164Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7590591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7591031Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7591489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7591914Z self_outputs = self.self( 2025-12-04T09:51:28.7592339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.7592850Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.7593053Z 2025-12-04T09:51:28.7593153Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7593386Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7593652Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7594081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7594420Z res = mod(**inputs) 2025-12-04T09:51:28.7594828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7595282Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7595735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7596206Z hidden_states = self.encoder( 2025-12-04T09:51:28.7596638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7597075Z layer_outputs = layer_module( 2025-12-04T09:51:28.7597441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7597841Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7598267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7598679Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7599143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7599585Z self_outputs = self.self( 2025-12-04T09:51:28.7600010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.7600491Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.7600748Z 2025-12-04T09:51:28.7600864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7601254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7601604Z res = mod(**inputs) 2025-12-04T09:51:28.7601994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7602415Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7602854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7604028Z hidden_states = self.encoder( 2025-12-04T09:51:28.7604419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7604828Z layer_outputs = layer_module( 2025-12-04T09:51:28.7605180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7605551Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7605952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7606371Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7606785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.7607304Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.7607775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.7608197Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7608413Z 2025-12-04T09:51:28.7608549Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7608916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7609245Z res = mod(**inputs) 2025-12-04T09:51:28.7609628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7610076Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7610482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7610890Z hidden_states = self.encoder( 2025-12-04T09:51:28.7611290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7611694Z layer_outputs = layer_module( 2025-12-04T09:51:28.7612049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7612418Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7612830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7613242Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7613654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7614054Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7614505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7614989Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7615449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.7615865Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7616007Z 2025-12-04T09:51:28.7616112Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7616477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7616806Z res = mod(**inputs) 2025-12-04T09:51:28.7617188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7617601Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7618082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7618510Z hidden_states = self.encoder( 2025-12-04T09:51:28.7618915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7619321Z layer_outputs = layer_module( 2025-12-04T09:51:28.7619688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7620085Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7620500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7620926Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7621338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7621759Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7622193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7622683Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7623144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.7623591Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.7623978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.7624377Z return self.act(input) 2025-12-04T09:51:28.7624500Z 2025-12-04T09:51:28.7624618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7625000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7625351Z res = mod(**inputs) 2025-12-04T09:51:28.7625761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7626218Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7626635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7627137Z hidden_states = self.encoder( 2025-12-04T09:51:28.7627564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7627997Z layer_outputs = layer_module( 2025-12-04T09:51:28.7628370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7628764Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7629206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7629650Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7630094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7630527Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7630998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.7631524Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.7632020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.7632471Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7632620Z 2025-12-04T09:51:28.7632740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7633147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7633515Z res = mod(**inputs) 2025-12-04T09:51:28.7633922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7634358Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7634799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7635240Z hidden_states = self.encoder( 2025-12-04T09:51:28.7635664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7636086Z layer_outputs = layer_module( 2025-12-04T09:51:28.7636506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7636910Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7637328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7637769Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7638207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7638643Z self_outputs = self.self( 2025-12-04T09:51:28.7639054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.7639531Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.7639689Z 2025-12-04T09:51:28.7639808Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7640193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7640614Z res = mod(**inputs) 2025-12-04T09:51:28.7641049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7641515Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7641963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7642396Z hidden_states = self.encoder( 2025-12-04T09:51:28.7642819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7643270Z layer_outputs = layer_module( 2025-12-04T09:51:28.7643633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7644004Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7644444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7644893Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7645300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7645710Z self_outputs = self.self( 2025-12-04T09:51:28.7646105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.7646519Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.7646670Z 2025-12-04T09:51:28.7646777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7647302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7647638Z res = mod(**inputs) 2025-12-04T09:51:28.7648015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7648508Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7648950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7649353Z hidden_states = self.encoder( 2025-12-04T09:51:28.7649749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7650160Z layer_outputs = layer_module( 2025-12-04T09:51:28.7650510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7650871Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7651286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7651709Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7652166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7652569Z self_outputs = self.self( 2025-12-04T09:51:28.7652961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.7653393Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.7653546Z 2025-12-04T09:51:28.7653632Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7653858Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7654102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7654511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7654834Z res = mod(**inputs) 2025-12-04T09:51:28.7655219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7655643Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7656058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7656475Z hidden_states = self.encoder( 2025-12-04T09:51:28.7656877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7657296Z layer_outputs = layer_module( 2025-12-04T09:51:28.7657663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7658056Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7658498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7658946Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7659375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7659784Z self_outputs = self.self( 2025-12-04T09:51:28.7660176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.7660624Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.7660801Z 2025-12-04T09:51:28.7660889Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7661145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7661526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7661865Z res = mod(**inputs) 2025-12-04T09:51:28.7662265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7662705Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7663158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7663605Z hidden_states = self.encoder( 2025-12-04T09:51:28.7664025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7664455Z layer_outputs = layer_module( 2025-12-04T09:51:28.7664821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7665209Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7665645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7666084Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7666535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7666965Z self_outputs = self.self( 2025-12-04T09:51:28.7667379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7667897Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7668418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.7668854Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.7668995Z 2025-12-04T09:51:28.7669135Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7669518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7669866Z res = mod(**inputs) 2025-12-04T09:51:28.7670271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7670714Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7671145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7671576Z hidden_states = self.encoder( 2025-12-04T09:51:28.7672006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7672405Z layer_outputs = layer_module( 2025-12-04T09:51:28.7672757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7673127Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7673539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7673951Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7674370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7674777Z self_outputs = self.self( 2025-12-04T09:51:28.7675161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7675655Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7676148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.7676556Z x = self.pointwise(x) 2025-12-04T09:51:28.7676670Z 2025-12-04T09:51:28.7676776Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7677143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7677471Z res = mod(**inputs) 2025-12-04T09:51:28.7677876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7678323Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7678763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7679194Z hidden_states = self.encoder( 2025-12-04T09:51:28.7679607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7680035Z layer_outputs = layer_module( 2025-12-04T09:51:28.7680407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7680868Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7681324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7681789Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7682222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7682627Z self_outputs = self.self( 2025-12-04T09:51:28.7683017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.7683512Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.7683721Z 2025-12-04T09:51:28.7683864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7684222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7684552Z res = mod(**inputs) 2025-12-04T09:51:28.7684936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7685357Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7685765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7686171Z hidden_states = self.encoder( 2025-12-04T09:51:28.7686567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7686974Z layer_outputs = layer_module( 2025-12-04T09:51:28.7687309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7687671Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7688073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7688473Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7688877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7689271Z self_outputs = self.self( 2025-12-04T09:51:28.7689654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.7690090Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.7690269Z 2025-12-04T09:51:28.7690375Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7690740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7691057Z res = mod(**inputs) 2025-12-04T09:51:28.7691422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7691827Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7692249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7692657Z hidden_states = self.encoder( 2025-12-04T09:51:28.7693047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7693444Z layer_outputs = layer_module( 2025-12-04T09:51:28.7693789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7694142Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7694547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7694951Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7695366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7695766Z self_outputs = self.self( 2025-12-04T09:51:28.7696160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.7696632Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.7696809Z 2025-12-04T09:51:28.7696890Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7697102Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7697334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7697684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7698015Z res = mod(**inputs) 2025-12-04T09:51:28.7698394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7698802Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7699207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7699609Z hidden_states = self.encoder( 2025-12-04T09:51:28.7700005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7700402Z layer_outputs = layer_module( 2025-12-04T09:51:28.7700745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7701115Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7701535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7701951Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7702372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7702785Z self_outputs = self.self( 2025-12-04T09:51:28.7703188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.7703636Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.7703813Z 2025-12-04T09:51:28.7703924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7704295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7704628Z res = mod(**inputs) 2025-12-04T09:51:28.7705006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7705434Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7705856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7706281Z hidden_states = self.encoder( 2025-12-04T09:51:28.7706711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7707123Z layer_outputs = layer_module( 2025-12-04T09:51:28.7707474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7707834Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7708246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7708664Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7709077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.7709531Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.7710010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.7710430Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7710570Z 2025-12-04T09:51:28.7710677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7711037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7711365Z res = mod(**inputs) 2025-12-04T09:51:28.7711742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7712208Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7712629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7713045Z hidden_states = self.encoder( 2025-12-04T09:51:28.7713451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7713866Z layer_outputs = layer_module( 2025-12-04T09:51:28.7714246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7714645Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7715078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7715518Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7715942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7716354Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7716799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7717301Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7717782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.7718232Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7718384Z 2025-12-04T09:51:28.7718501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7718896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7719252Z res = mod(**inputs) 2025-12-04T09:51:28.7719658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7720155Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7720716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7721204Z hidden_states = self.encoder( 2025-12-04T09:51:28.7721661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7722122Z layer_outputs = layer_module( 2025-12-04T09:51:28.7722499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7722887Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7723331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7723784Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7724224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7724685Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7725172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7725697Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7726186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.7726661Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.7727083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.7727490Z return self.act(input) 2025-12-04T09:51:28.7727613Z 2025-12-04T09:51:28.7727732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7728109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7728456Z res = mod(**inputs) 2025-12-04T09:51:28.7728909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7729347Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7729789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7730275Z hidden_states = self.encoder( 2025-12-04T09:51:28.7730703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7731149Z layer_outputs = layer_module( 2025-12-04T09:51:28.7731543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7731948Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7732398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7732845Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7733285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7733713Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7734171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.7734700Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.7735197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.7735637Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7735787Z 2025-12-04T09:51:28.7735898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7736284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7736657Z res = mod(**inputs) 2025-12-04T09:51:28.7737072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7737517Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7737814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7737892Z hidden_states = self.encoder( 2025-12-04T09:51:28.7738181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7738268Z layer_outputs = layer_module( 2025-12-04T09:51:28.7738509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7738594Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7738910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7739003Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7739298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7739376Z self_outputs = self.self( 2025-12-04T09:51:28.7739662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.7739770Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.7739792Z 2025-12-04T09:51:28.7739906Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7740125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7740196Z res = mod(**inputs) 2025-12-04T09:51:28.7740487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7740584Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7740874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7740950Z hidden_states = self.encoder( 2025-12-04T09:51:28.7741247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7741323Z layer_outputs = layer_module( 2025-12-04T09:51:28.7741573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7741656Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7741947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7742043Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7742336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7742421Z self_outputs = self.self( 2025-12-04T09:51:28.7742707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.7742796Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.7742800Z 2025-12-04T09:51:28.7742919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7743134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7743204Z res = mod(**inputs) 2025-12-04T09:51:28.7743502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7743589Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7743912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7744008Z hidden_states = self.encoder( 2025-12-04T09:51:28.7744291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7744374Z layer_outputs = layer_module( 2025-12-04T09:51:28.7744611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7744704Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7744988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7745076Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7745394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7745477Z self_outputs = self.self( 2025-12-04T09:51:28.7745769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.7745877Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.7745881Z 2025-12-04T09:51:28.7745969Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7746060Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7746171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7746408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7746487Z res = mod(**inputs) 2025-12-04T09:51:28.7746776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7746861Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7747341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7747425Z hidden_states = self.encoder( 2025-12-04T09:51:28.7747725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7747802Z layer_outputs = layer_module( 2025-12-04T09:51:28.7748045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7748138Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7748427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7748523Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7748809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7748889Z self_outputs = self.self( 2025-12-04T09:51:28.7749184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.7749298Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.7749302Z 2025-12-04T09:51:28.7749390Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7749511Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7749723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7749803Z res = mod(**inputs) 2025-12-04T09:51:28.7750092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7750178Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7750540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7750644Z hidden_states = self.encoder( 2025-12-04T09:51:28.7750939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7751015Z layer_outputs = layer_module( 2025-12-04T09:51:28.7751254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7751343Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7751630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7751719Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7752015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7752116Z self_outputs = self.self( 2025-12-04T09:51:28.7752420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7752598Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7752888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.7752983Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.7752987Z 2025-12-04T09:51:28.7753099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7753347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7753416Z res = mod(**inputs) 2025-12-04T09:51:28.7753701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7753796Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7754086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7754162Z hidden_states = self.encoder( 2025-12-04T09:51:28.7754453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7754528Z layer_outputs = layer_module( 2025-12-04T09:51:28.7754772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7754855Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7755140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7755233Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7755523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7755609Z self_outputs = self.self( 2025-12-04T09:51:28.7755898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7756073Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7756346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.7756420Z x = self.pointwise(x) 2025-12-04T09:51:28.7756425Z 2025-12-04T09:51:28.7756530Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7756750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7756821Z res = mod(**inputs) 2025-12-04T09:51:28.7757138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7757243Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7757531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7757614Z hidden_states = self.encoder( 2025-12-04T09:51:28.7757904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7757990Z layer_outputs = layer_module( 2025-12-04T09:51:28.7758232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7758313Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7758607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7758715Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7759007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7759093Z self_outputs = self.self( 2025-12-04T09:51:28.7759382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.7759556Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.7759560Z 2025-12-04T09:51:28.7759670Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7759911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7759991Z res = mod(**inputs) 2025-12-04T09:51:28.7760277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7760373Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7760722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7760806Z hidden_states = self.encoder( 2025-12-04T09:51:28.7761103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7761180Z layer_outputs = layer_module( 2025-12-04T09:51:28.7761424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7761517Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7761806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7761900Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7762191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7762270Z self_outputs = self.self( 2025-12-04T09:51:28.7762566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.7762687Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.7762691Z 2025-12-04T09:51:28.7762807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7763009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7763089Z res = mod(**inputs) 2025-12-04T09:51:28.7763376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7763460Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7763763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7763872Z hidden_states = self.encoder( 2025-12-04T09:51:28.7764144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7764224Z layer_outputs = layer_module( 2025-12-04T09:51:28.7764454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7764531Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7764810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7764893Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7765173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7765262Z self_outputs = self.self( 2025-12-04T09:51:28.7765535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.7765673Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.7765677Z 2025-12-04T09:51:28.7765759Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7765842Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7765956Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7766155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7766248Z res = mod(**inputs) 2025-12-04T09:51:28.7766527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7766610Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7766899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7766975Z hidden_states = self.encoder( 2025-12-04T09:51:28.7767253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7767333Z layer_outputs = layer_module( 2025-12-04T09:51:28.7767564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7767650Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7767930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7768017Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7768299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7768371Z self_outputs = self.self( 2025-12-04T09:51:28.7768659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.7768777Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.7768781Z 2025-12-04T09:51:28.7768890Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7769103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7769170Z res = mod(**inputs) 2025-12-04T09:51:28.7769447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7769537Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7769812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7769891Z hidden_states = self.encoder( 2025-12-04T09:51:28.7770187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7770274Z layer_outputs = layer_module( 2025-12-04T09:51:28.7770511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7770587Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7770868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7770952Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7771232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.7771371Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.7771664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.7771752Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7771762Z 2025-12-04T09:51:28.7771866Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7772066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7772138Z res = mod(**inputs) 2025-12-04T09:51:28.7772408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7772518Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7772796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7772868Z hidden_states = self.encoder( 2025-12-04T09:51:28.7773148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7773223Z layer_outputs = layer_module( 2025-12-04T09:51:28.7773450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7773536Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7773802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7773891Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7774164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7774247Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7774559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7774684Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7774956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.7775047Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7775051Z 2025-12-04T09:51:28.7775157Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7775362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7775429Z res = mod(**inputs) 2025-12-04T09:51:28.7775704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7775796Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7776060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7776132Z hidden_states = self.encoder( 2025-12-04T09:51:28.7776451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7776539Z layer_outputs = layer_module( 2025-12-04T09:51:28.7776776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7776852Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7777129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7777221Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7777491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7777576Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7777904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7778030Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7778300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.7778410Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.7778622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.7778701Z return self.act(input) 2025-12-04T09:51:28.7778725Z 2025-12-04T09:51:28.7778828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7779034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7779104Z res = mod(**inputs) 2025-12-04T09:51:28.7779397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7779497Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7779793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7779873Z hidden_states = self.encoder( 2025-12-04T09:51:28.7780144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7780217Z layer_outputs = layer_module( 2025-12-04T09:51:28.7780451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7780535Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7780806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7780898Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7781167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7781254Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7781561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.7781698Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.7781977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.7782063Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7782066Z 2025-12-04T09:51:28.7782177Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7782378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7782445Z res = mod(**inputs) 2025-12-04T09:51:28.7782747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7782845Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7783118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7783200Z hidden_states = self.encoder( 2025-12-04T09:51:28.7783480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7783559Z layer_outputs = layer_module( 2025-12-04T09:51:28.7783780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7783856Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7784150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7784236Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7784513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7784584Z self_outputs = self.self( 2025-12-04T09:51:28.7784870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.7784974Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.7784978Z 2025-12-04T09:51:28.7785149Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7785362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7785438Z res = mod(**inputs) 2025-12-04T09:51:28.7785727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7785822Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7786111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7786189Z hidden_states = self.encoder( 2025-12-04T09:51:28.7786484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7786560Z layer_outputs = layer_module( 2025-12-04T09:51:28.7786805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7786889Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7787183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7787273Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7787546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7787619Z self_outputs = self.self( 2025-12-04T09:51:28.7787895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.7787980Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.7787984Z 2025-12-04T09:51:28.7788093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7788291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7788358Z res = mod(**inputs) 2025-12-04T09:51:28.7788636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7788718Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7789020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7789117Z hidden_states = self.encoder( 2025-12-04T09:51:28.7789403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7789486Z layer_outputs = layer_module( 2025-12-04T09:51:28.7789722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7789806Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7790097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7790183Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7790478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7790566Z self_outputs = self.self( 2025-12-04T09:51:28.7790845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.7790946Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.7790949Z 2025-12-04T09:51:28.7791032Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7791120Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7791225Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7791422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7791511Z res = mod(**inputs) 2025-12-04T09:51:28.7791785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7791869Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7792153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7792228Z hidden_states = self.encoder( 2025-12-04T09:51:28.7792511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7792582Z layer_outputs = layer_module( 2025-12-04T09:51:28.7792808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7792895Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7793168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7793251Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7793529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7793603Z self_outputs = self.self( 2025-12-04T09:51:28.7793885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.7793988Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.7793992Z 2025-12-04T09:51:28.7794072Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7794185Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7794386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7794459Z res = mod(**inputs) 2025-12-04T09:51:28.7794734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7794816Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7795096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7795184Z hidden_states = self.encoder( 2025-12-04T09:51:28.7795484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7795563Z layer_outputs = layer_module( 2025-12-04T09:51:28.7795798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7795885Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7796163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7796246Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7796525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7796596Z self_outputs = self.self( 2025-12-04T09:51:28.7796910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7797078Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7797348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.7797435Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.7797438Z 2025-12-04T09:51:28.7797545Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7797756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7797855Z res = mod(**inputs) 2025-12-04T09:51:28.7798143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7798238Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7798528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7798606Z hidden_states = self.encoder( 2025-12-04T09:51:28.7798900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7798978Z layer_outputs = layer_module( 2025-12-04T09:51:28.7799222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7799303Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7799593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7799687Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7799975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7800053Z self_outputs = self.self( 2025-12-04T09:51:28.7800344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7800514Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7800900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.7800987Z x = self.pointwise(x) 2025-12-04T09:51:28.7800991Z 2025-12-04T09:51:28.7801110Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7801336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7801408Z res = mod(**inputs) 2025-12-04T09:51:28.7801727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7801837Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7802158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7802244Z hidden_states = self.encoder( 2025-12-04T09:51:28.7802532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7802610Z layer_outputs = layer_module( 2025-12-04T09:51:28.7802866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7802944Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7803229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7803311Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7803601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7803684Z self_outputs = self.self( 2025-12-04T09:51:28.7803954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.7804117Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.7804121Z 2025-12-04T09:51:28.7804225Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7804425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7804517Z res = mod(**inputs) 2025-12-04T09:51:28.7804806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7804895Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7805202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7805276Z hidden_states = self.encoder( 2025-12-04T09:51:28.7805556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7805628Z layer_outputs = layer_module( 2025-12-04T09:51:28.7805854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7805940Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7806216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7806306Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7806577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7806650Z self_outputs = self.self( 2025-12-04T09:51:28.7806930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.7807051Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.7807055Z 2025-12-04T09:51:28.7807160Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7807372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7807440Z res = mod(**inputs) 2025-12-04T09:51:28.7807720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7807802Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7808075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7808175Z hidden_states = self.encoder( 2025-12-04T09:51:28.7808463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7808542Z layer_outputs = layer_module( 2025-12-04T09:51:28.7808768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7808846Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7809124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7809208Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7809480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7809558Z self_outputs = self.self( 2025-12-04T09:51:28.7809844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.7809984Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.7809988Z 2025-12-04T09:51:28.7810070Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7810151Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7810261Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7810463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7810528Z res = mod(**inputs) 2025-12-04T09:51:28.7810828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7810909Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7811188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7811266Z hidden_states = self.encoder( 2025-12-04T09:51:28.7811552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7811637Z layer_outputs = layer_module( 2025-12-04T09:51:28.7811874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7811963Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7812249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7812337Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7812629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7812704Z self_outputs = self.self( 2025-12-04T09:51:28.7812992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.7813122Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.7813126Z 2025-12-04T09:51:28.7813244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7813451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7813517Z res = mod(**inputs) 2025-12-04T09:51:28.7813789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7813885Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7814169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7814253Z hidden_states = self.encoder( 2025-12-04T09:51:28.7814556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7814651Z layer_outputs = layer_module( 2025-12-04T09:51:28.7814904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7814987Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7815286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7815379Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7815680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.7815825Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.7816146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.7816237Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7816242Z 2025-12-04T09:51:28.7816360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7816572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7816647Z res = mod(**inputs) 2025-12-04T09:51:28.7816933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7817021Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7817331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7817406Z hidden_states = self.encoder( 2025-12-04T09:51:28.7817690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7817773Z layer_outputs = layer_module( 2025-12-04T09:51:28.7818012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7818102Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7818387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7818478Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7818763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7818849Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7819174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7819303Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7819592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.7819689Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7819692Z 2025-12-04T09:51:28.7819801Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7820018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7820087Z res = mod(**inputs) 2025-12-04T09:51:28.7820375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7820468Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7820752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7820829Z hidden_states = self.encoder( 2025-12-04T09:51:28.7821143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7821268Z layer_outputs = layer_module( 2025-12-04T09:51:28.7821517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7821600Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7821891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7821988Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7822276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7822358Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7822708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7822843Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7823150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.7823265Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.7823484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.7823566Z return self.act(input) 2025-12-04T09:51:28.7823570Z 2025-12-04T09:51:28.7823675Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7823910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7823977Z res = mod(**inputs) 2025-12-04T09:51:28.7824245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7824340Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7824614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7824688Z hidden_states = self.encoder( 2025-12-04T09:51:28.7824977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7825056Z layer_outputs = layer_module( 2025-12-04T09:51:28.7825304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7825389Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7825675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7825771Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7826051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7826150Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7826451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.7826585Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.7826865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.7826950Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7826954Z 2025-12-04T09:51:28.7827065Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7827266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7827332Z res = mod(**inputs) 2025-12-04T09:51:28.7827636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7827735Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7828013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7828098Z hidden_states = self.encoder( 2025-12-04T09:51:28.7828384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7828468Z layer_outputs = layer_module( 2025-12-04T09:51:28.7828709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7828793Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7829090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7829195Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7829489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7829572Z self_outputs = self.self( 2025-12-04T09:51:28.7829861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.7829968Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.7829972Z 2025-12-04T09:51:28.7830084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7830316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7830395Z res = mod(**inputs) 2025-12-04T09:51:28.7830685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7830781Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7831070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7831147Z hidden_states = self.encoder( 2025-12-04T09:51:28.7831440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7831517Z layer_outputs = layer_module( 2025-12-04T09:51:28.7831755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7831846Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7832132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7832225Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7832513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7832590Z self_outputs = self.self( 2025-12-04T09:51:28.7832906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.7832994Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.7832998Z 2025-12-04T09:51:28.7833115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7833327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7833398Z res = mod(**inputs) 2025-12-04T09:51:28.7833692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7833778Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7834063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7834186Z hidden_states = self.encoder( 2025-12-04T09:51:28.7834472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7834555Z layer_outputs = layer_module( 2025-12-04T09:51:28.7834789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7834872Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7835171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7835260Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7835554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7835629Z self_outputs = self.self( 2025-12-04T09:51:28.7835935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.7836044Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.7836047Z 2025-12-04T09:51:28.7836133Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7836217Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7836334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7836544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7836643Z res = mod(**inputs) 2025-12-04T09:51:28.7836936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7837023Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7837324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7837403Z hidden_states = self.encoder( 2025-12-04T09:51:28.7837696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7837778Z layer_outputs = layer_module( 2025-12-04T09:51:28.7838019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7838108Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7838401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7838490Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7838792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7838868Z self_outputs = self.self( 2025-12-04T09:51:28.7839169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.7839283Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.7839287Z 2025-12-04T09:51:28.7839371Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7839490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7839708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7839778Z res = mod(**inputs) 2025-12-04T09:51:28.7840083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7840174Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7840488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7840647Z hidden_states = self.encoder( 2025-12-04T09:51:28.7840997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7841088Z layer_outputs = layer_module( 2025-12-04T09:51:28.7841333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7841427Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7841722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7841813Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7842126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7842202Z self_outputs = self.self( 2025-12-04T09:51:28.7842512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7842702Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7842990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.7843082Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.7843086Z 2025-12-04T09:51:28.7843199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7843410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7843509Z res = mod(**inputs) 2025-12-04T09:51:28.7843799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7843892Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7844182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7844260Z hidden_states = self.encoder( 2025-12-04T09:51:28.7844554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7844630Z layer_outputs = layer_module( 2025-12-04T09:51:28.7844869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7844959Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7845247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7845345Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7845636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7845714Z self_outputs = self.self( 2025-12-04T09:51:28.7846016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7846184Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7846479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.7846557Z x = self.pointwise(x) 2025-12-04T09:51:28.7846561Z 2025-12-04T09:51:28.7846672Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7846894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7846964Z res = mod(**inputs) 2025-12-04T09:51:28.7847412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7847517Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7847902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7847991Z hidden_states = self.encoder( 2025-12-04T09:51:28.7848277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7848354Z layer_outputs = layer_module( 2025-12-04T09:51:28.7848600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7848683Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7848975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7849062Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7849380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7849469Z self_outputs = self.self( 2025-12-04T09:51:28.7849756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.7849922Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.7849934Z 2025-12-04T09:51:28.7850045Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7850255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7850363Z res = mod(**inputs) 2025-12-04T09:51:28.7850646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7850733Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7851027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7851105Z hidden_states = self.encoder( 2025-12-04T09:51:28.7851404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7851481Z layer_outputs = layer_module( 2025-12-04T09:51:28.7851715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7851807Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7852092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7852179Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7852475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7852550Z self_outputs = self.self( 2025-12-04T09:51:28.7852846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.7852978Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.7852982Z 2025-12-04T09:51:28.7853090Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7853310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7853381Z res = mod(**inputs) 2025-12-04T09:51:28.7853669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7853757Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7854040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7854125Z hidden_states = self.encoder( 2025-12-04T09:51:28.7854428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7854523Z layer_outputs = layer_module( 2025-12-04T09:51:28.7854770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7854853Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7855152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7855240Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7855530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7855612Z self_outputs = self.self( 2025-12-04T09:51:28.7855975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.7856116Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.7856120Z 2025-12-04T09:51:28.7856203Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7856286Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7856400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7856608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7856681Z res = mod(**inputs) 2025-12-04T09:51:28.7856980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7857088Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7857383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7857461Z hidden_states = self.encoder( 2025-12-04T09:51:28.7857749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7857838Z layer_outputs = layer_module( 2025-12-04T09:51:28.7858077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7858158Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7858453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7858542Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7858832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7858903Z self_outputs = self.self( 2025-12-04T09:51:28.7859175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.7859300Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.7859304Z 2025-12-04T09:51:28.7859409Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7859619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7859684Z res = mod(**inputs) 2025-12-04T09:51:28.7859953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7860043Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7860312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7860384Z hidden_states = self.encoder( 2025-12-04T09:51:28.7860680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7860793Z layer_outputs = layer_module( 2025-12-04T09:51:28.7861039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7861122Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7861412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7861510Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7861801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.7861951Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.7862237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.7862351Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7862359Z 2025-12-04T09:51:28.7862477Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7862686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7862756Z res = mod(**inputs) 2025-12-04T09:51:28.7863049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7863135Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7863430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7863528Z hidden_states = self.encoder( 2025-12-04T09:51:28.7863814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7863898Z layer_outputs = layer_module( 2025-12-04T09:51:28.7864137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7864229Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7864513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7864605Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7864891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7864977Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7865295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7865433Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7865719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.7865816Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7865819Z 2025-12-04T09:51:28.7865929Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7866148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7866222Z res = mod(**inputs) 2025-12-04T09:51:28.7866490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7866580Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7866854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7866936Z hidden_states = self.encoder( 2025-12-04T09:51:28.7867229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7867344Z layer_outputs = layer_module( 2025-12-04T09:51:28.7867583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7867676Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7867962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7868057Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7868334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7868419Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7868752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7868894Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7869174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.7869287Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.7869505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.7869585Z return self.act(input) 2025-12-04T09:51:28.7869589Z 2025-12-04T09:51:28.7869692Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7869899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7869999Z res = mod(**inputs) 2025-12-04T09:51:28.7870288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7870382Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7870673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7870752Z hidden_states = self.encoder( 2025-12-04T09:51:28.7871049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7871125Z layer_outputs = layer_module( 2025-12-04T09:51:28.7871370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7871455Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7871748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7871845Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7872127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7872212Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7872545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.7872678Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.7872962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.7873050Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7873055Z 2025-12-04T09:51:28.7873163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7873383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7873453Z res = mod(**inputs) 2025-12-04T09:51:28.7873752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7873875Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7874165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7874250Z hidden_states = self.encoder( 2025-12-04T09:51:28.7874540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7874625Z layer_outputs = layer_module( 2025-12-04T09:51:28.7874865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7874950Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7875245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7875332Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7875653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7875739Z self_outputs = self.self( 2025-12-04T09:51:28.7876026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.7876130Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.7876134Z 2025-12-04T09:51:28.7876243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7876452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7876550Z res = mod(**inputs) 2025-12-04T09:51:28.7876848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7876938Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7877248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7877327Z hidden_states = self.encoder( 2025-12-04T09:51:28.7877634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7877710Z layer_outputs = layer_module( 2025-12-04T09:51:28.7877960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7878050Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7878349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7878443Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7878742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7878820Z self_outputs = self.self( 2025-12-04T09:51:28.7879128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.7879217Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.7879221Z 2025-12-04T09:51:28.7879342Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7879566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7879638Z res = mod(**inputs) 2025-12-04T09:51:28.7879956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7880044Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7880353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7880441Z hidden_states = self.encoder( 2025-12-04T09:51:28.7880859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7880953Z layer_outputs = layer_module( 2025-12-04T09:51:28.7881201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7881286Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7881594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7881684Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7881985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7882069Z self_outputs = self.self( 2025-12-04T09:51:28.7882374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.7882485Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.7882489Z 2025-12-04T09:51:28.7882576Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7882664Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7882783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7883004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7883078Z res = mod(**inputs) 2025-12-04T09:51:28.7883348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7883450Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7883729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7883804Z hidden_states = self.encoder( 2025-12-04T09:51:28.7884093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7884176Z layer_outputs = layer_module( 2025-12-04T09:51:28.7884409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7884498Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7884775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7884860Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7885137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7885580Z self_outputs = self.self( 2025-12-04T09:51:28.7886000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.7886476Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.7886641Z 2025-12-04T09:51:28.7886731Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7886991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7887392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7887765Z res = mod(**inputs) 2025-12-04T09:51:28.7888192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7888650Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7889113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7889563Z hidden_states = self.encoder( 2025-12-04T09:51:28.7890021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7890486Z layer_outputs = layer_module( 2025-12-04T09:51:28.7890870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7891282Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7891716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7892172Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7892619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7893116Z self_outputs = self.self( 2025-12-04T09:51:28.7893597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7894131Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7894642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.7895063Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.7895197Z 2025-12-04T09:51:28.7895303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7895676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7896011Z res = mod(**inputs) 2025-12-04T09:51:28.7896416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7896845Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7897270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7897687Z hidden_states = self.encoder( 2025-12-04T09:51:28.7898090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7898504Z layer_outputs = layer_module( 2025-12-04T09:51:28.7898873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7899272Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7899708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7900151Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7900595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7901027Z self_outputs = self.self( 2025-12-04T09:51:28.7901452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.7901980Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.7902511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.7902939Z x = self.pointwise(x) 2025-12-04T09:51:28.7903069Z 2025-12-04T09:51:28.7903182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7903571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7903924Z res = mod(**inputs) 2025-12-04T09:51:28.7904330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7904776Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7905244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7905692Z hidden_states = self.encoder( 2025-12-04T09:51:28.7906117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7906552Z layer_outputs = layer_module( 2025-12-04T09:51:28.7906927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7907312Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7907755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7908203Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7908666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7909103Z self_outputs = self.self( 2025-12-04T09:51:28.7909524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.7910044Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.7910267Z 2025-12-04T09:51:28.7910380Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7910772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7911116Z res = mod(**inputs) 2025-12-04T09:51:28.7911551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7911984Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7912425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7912862Z hidden_states = self.encoder( 2025-12-04T09:51:28.7913277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7913706Z layer_outputs = layer_module( 2025-12-04T09:51:28.7914082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7914473Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7914902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7915351Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7915791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7916222Z self_outputs = self.self( 2025-12-04T09:51:28.7916635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.7917119Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.7917305Z 2025-12-04T09:51:28.7917426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7917803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7918146Z res = mod(**inputs) 2025-12-04T09:51:28.7918547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7918990Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7919417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7919849Z hidden_states = self.encoder( 2025-12-04T09:51:28.7920296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7920993Z layer_outputs = layer_module( 2025-12-04T09:51:28.7921382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7921795Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7922252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7922777Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7923241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7923702Z self_outputs = self.self( 2025-12-04T09:51:28.7924147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.7924692Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.7924904Z 2025-12-04T09:51:28.7924997Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7925249Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.7925516Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7925928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7926308Z res = mod(**inputs) 2025-12-04T09:51:28.7926739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7927232Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7927693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7928145Z hidden_states = self.encoder( 2025-12-04T09:51:28.7928588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7929040Z layer_outputs = layer_module( 2025-12-04T09:51:28.7929429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7929852Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7930302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7930762Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7931221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7931662Z self_outputs = self.self( 2025-12-04T09:51:28.7932087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.7932588Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.7932774Z 2025-12-04T09:51:28.7932901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7933298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7933659Z res = mod(**inputs) 2025-12-04T09:51:28.7934087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7934546Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7935000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7935452Z hidden_states = self.encoder( 2025-12-04T09:51:28.7935880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7936320Z layer_outputs = layer_module( 2025-12-04T09:51:28.7936747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7937138Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7937580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7938015Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7938455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.7938947Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.7939441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.7939884Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7940041Z 2025-12-04T09:51:28.7940181Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7940578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7940929Z res = mod(**inputs) 2025-12-04T09:51:28.7941340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7941794Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7942251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7942707Z hidden_states = self.encoder( 2025-12-04T09:51:28.7943140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7943577Z layer_outputs = layer_module( 2025-12-04T09:51:28.7943967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7944362Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7944805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7945257Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7945713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7946145Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7946625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7947347Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7947847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.7948303Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7948466Z 2025-12-04T09:51:28.7948580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7948976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7949332Z res = mod(**inputs) 2025-12-04T09:51:28.7949753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7950214Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7950682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7951119Z hidden_states = self.encoder( 2025-12-04T09:51:28.7951552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7952000Z layer_outputs = layer_module( 2025-12-04T09:51:28.7952447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7952847Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7953304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7953770Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7954209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7954652Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7955101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.7955589Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.7956089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.7956549Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.7956946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.7957322Z return self.act(input) 2025-12-04T09:51:28.7957451Z 2025-12-04T09:51:28.7957562Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7957961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7958339Z res = mod(**inputs) 2025-12-04T09:51:28.7958753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7959204Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7959655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7960083Z hidden_states = self.encoder( 2025-12-04T09:51:28.7960512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7961028Z layer_outputs = layer_module( 2025-12-04T09:51:28.7961421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7961845Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7962289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.7962789Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.7963201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.7963608Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.7964056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.7964562Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.7965025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.7965445Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.7965596Z 2025-12-04T09:51:28.7965703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7966077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7966406Z res = mod(**inputs) 2025-12-04T09:51:28.7966802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7967258Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7967731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7968236Z hidden_states = self.encoder( 2025-12-04T09:51:28.7968639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7969075Z layer_outputs = layer_module( 2025-12-04T09:51:28.7969446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7969846Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7970286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7970735Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7971200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7971649Z self_outputs = self.self( 2025-12-04T09:51:28.7972075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.7972530Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.7972698Z 2025-12-04T09:51:28.7972813Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7973202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7973577Z res = mod(**inputs) 2025-12-04T09:51:28.7973974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7974418Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7974863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7975287Z hidden_states = self.encoder( 2025-12-04T09:51:28.7975710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.7976142Z layer_outputs = layer_module( 2025-12-04T09:51:28.7976510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.7976862Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.7977262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.7994710Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.7995263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.7995752Z self_outputs = self.self( 2025-12-04T09:51:28.7996213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.7996685Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.7996861Z 2025-12-04T09:51:28.7996986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.7997395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.7997771Z res = mod(**inputs) 2025-12-04T09:51:28.7998197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.7998668Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.7999139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.7999602Z hidden_states = self.encoder( 2025-12-04T09:51:28.8000158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8000756Z layer_outputs = layer_module( 2025-12-04T09:51:28.8001167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8001581Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8002062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8002529Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8002983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8003428Z self_outputs = self.self( 2025-12-04T09:51:28.8003874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.8004310Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.8004465Z 2025-12-04T09:51:28.8004566Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8004800Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8005061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8005452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8005781Z res = mod(**inputs) 2025-12-04T09:51:28.8006175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8006639Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8007091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8007534Z hidden_states = self.encoder( 2025-12-04T09:51:28.8007980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8008414Z layer_outputs = layer_module( 2025-12-04T09:51:28.8008791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8009194Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8009613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8010061Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8010501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8010933Z self_outputs = self.self( 2025-12-04T09:51:28.8011359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.8011831Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.8012004Z 2025-12-04T09:51:28.8012093Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8012358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8012756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8013103Z res = mod(**inputs) 2025-12-04T09:51:28.8013511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8013964Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8014409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8014836Z hidden_states = self.encoder( 2025-12-04T09:51:28.8015723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8016185Z layer_outputs = layer_module( 2025-12-04T09:51:28.8016556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8016954Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8017393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8017837Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8018269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8018699Z self_outputs = self.self( 2025-12-04T09:51:28.8019117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.8019672Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.8020218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.8020674Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.8020829Z 2025-12-04T09:51:28.8020951Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8021336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8021685Z res = mod(**inputs) 2025-12-04T09:51:28.8022088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8022554Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8023004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8023436Z hidden_states = self.encoder( 2025-12-04T09:51:28.8023861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8024291Z layer_outputs = layer_module( 2025-12-04T09:51:28.8024661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8025054Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8025499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8025934Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8026389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8026828Z self_outputs = self.self( 2025-12-04T09:51:28.8027273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.8027799Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.8028335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.8028775Z x = self.pointwise(x) 2025-12-04T09:51:28.8028897Z 2025-12-04T09:51:28.8029015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8029398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8029751Z res = mod(**inputs) 2025-12-04T09:51:28.8030165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8030610Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8031084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8031536Z hidden_states = self.encoder( 2025-12-04T09:51:28.8031967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8032392Z layer_outputs = layer_module( 2025-12-04T09:51:28.8032764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8033157Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8033588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8034039Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8034498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8034964Z self_outputs = self.self( 2025-12-04T09:51:28.8035396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.8035920Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.8036143Z 2025-12-04T09:51:28.8036262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8036652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8036994Z res = mod(**inputs) 2025-12-04T09:51:28.8037397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8037866Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8038304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8038749Z hidden_states = self.encoder( 2025-12-04T09:51:28.8039184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8039633Z layer_outputs = layer_module( 2025-12-04T09:51:28.8040012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8040417Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8040966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8041432Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8041884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8042333Z self_outputs = self.self( 2025-12-04T09:51:28.8042767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.8043264Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.8043466Z 2025-12-04T09:51:28.8043587Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8043986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8044346Z res = mod(**inputs) 2025-12-04T09:51:28.8044772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8045236Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8045692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8046154Z hidden_states = self.encoder( 2025-12-04T09:51:28.8046598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8047335Z layer_outputs = layer_module( 2025-12-04T09:51:28.8047738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8048134Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8048598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8049069Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8049535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8049976Z self_outputs = self.self( 2025-12-04T09:51:28.8050422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.8050990Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.8051193Z 2025-12-04T09:51:28.8051293Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8051530Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8051795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8052197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8052545Z res = mod(**inputs) 2025-12-04T09:51:28.8052971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8053448Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8053889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8054321Z hidden_states = self.encoder( 2025-12-04T09:51:28.8054745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8055180Z layer_outputs = layer_module( 2025-12-04T09:51:28.8055548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8055938Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8056370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8056811Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8057239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8057671Z self_outputs = self.self( 2025-12-04T09:51:28.8058085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.8058551Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.8058738Z 2025-12-04T09:51:28.8058848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8059233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8059580Z res = mod(**inputs) 2025-12-04T09:51:28.8059978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8060415Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8060852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8061281Z hidden_states = self.encoder( 2025-12-04T09:51:28.8061690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8062119Z layer_outputs = layer_module( 2025-12-04T09:51:28.8062518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8062933Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8063349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8063788Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8064229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.8064719Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.8065213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.8065654Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8065797Z 2025-12-04T09:51:28.8065932Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8066307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8066646Z res = mod(**inputs) 2025-12-04T09:51:28.8067037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8067486Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8067945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8068410Z hidden_states = self.encoder( 2025-12-04T09:51:28.8068808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8069216Z layer_outputs = layer_module( 2025-12-04T09:51:28.8069573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8069962Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8070393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8070855Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8071295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8071725Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8072191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.8072714Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.8073203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.8073646Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8073805Z 2025-12-04T09:51:28.8073919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8074310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8074671Z res = mod(**inputs) 2025-12-04T09:51:28.8075049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8075472Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8075890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8076322Z hidden_states = self.encoder( 2025-12-04T09:51:28.8076740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8077171Z layer_outputs = layer_module( 2025-12-04T09:51:28.8077573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8077983Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8078419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8078867Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8079308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8079730Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8080196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.8080793Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.8081319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.8081808Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.8082237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.8082623Z return self.act(input) 2025-12-04T09:51:28.8082749Z 2025-12-04T09:51:28.8082866Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8083270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8083643Z res = mod(**inputs) 2025-12-04T09:51:28.8084051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8084487Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8084931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8085366Z hidden_states = self.encoder( 2025-12-04T09:51:28.8085786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8086209Z layer_outputs = layer_module( 2025-12-04T09:51:28.8086583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8086971Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8087400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8087850Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8088283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8088704Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8089162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.8089692Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.8090186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.8090623Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8090774Z 2025-12-04T09:51:28.8090886Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8091278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8091623Z res = mod(**inputs) 2025-12-04T09:51:28.8091987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8092400Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8092848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8093303Z hidden_states = self.encoder( 2025-12-04T09:51:28.8093716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8094144Z layer_outputs = layer_module( 2025-12-04T09:51:28.8094497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8094858Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8095259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8095662Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8096097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8096504Z self_outputs = self.self( 2025-12-04T09:51:28.8096902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.8097333Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.8097485Z 2025-12-04T09:51:28.8097601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8097959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8098286Z res = mod(**inputs) 2025-12-04T09:51:28.8098692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8099102Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8099523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8099935Z hidden_states = self.encoder( 2025-12-04T09:51:28.8100333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8100746Z layer_outputs = layer_module( 2025-12-04T09:51:28.8101092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8101452Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8101850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8102252Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8102657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8103055Z self_outputs = self.self( 2025-12-04T09:51:28.8103433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.8103842Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.8103988Z 2025-12-04T09:51:28.8104091Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8104447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8104757Z res = mod(**inputs) 2025-12-04T09:51:28.8105130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8105537Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8105936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8106337Z hidden_states = self.encoder( 2025-12-04T09:51:28.8106748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8107173Z layer_outputs = layer_module( 2025-12-04T09:51:28.8107506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8107865Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8108258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8108665Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8109067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8109475Z self_outputs = self.self( 2025-12-04T09:51:28.8109869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.8110297Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.8110461Z 2025-12-04T09:51:28.8110545Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8110762Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8110997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8111348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8111669Z res = mod(**inputs) 2025-12-04T09:51:28.8112044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8112470Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8112748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8112821Z hidden_states = self.encoder( 2025-12-04T09:51:28.8113101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8113175Z layer_outputs = layer_module( 2025-12-04T09:51:28.8113405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8113492Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8113772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8113862Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8114135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8114209Z self_outputs = self.self( 2025-12-04T09:51:28.8114490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.8114595Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.8114600Z 2025-12-04T09:51:28.8114682Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8114795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8115000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8115075Z res = mod(**inputs) 2025-12-04T09:51:28.8115353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8115434Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8115723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8115795Z hidden_states = self.encoder( 2025-12-04T09:51:28.8116071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8116163Z layer_outputs = layer_module( 2025-12-04T09:51:28.8116407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8116493Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8116765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8116848Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8117125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8117201Z self_outputs = self.self( 2025-12-04T09:51:28.8117478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.8117643Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.8117935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.8118028Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.8118032Z 2025-12-04T09:51:28.8118138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8118349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8118417Z res = mod(**inputs) 2025-12-04T09:51:28.8118741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8118856Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8119145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8119222Z hidden_states = self.encoder( 2025-12-04T09:51:28.8119522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8119599Z layer_outputs = layer_module( 2025-12-04T09:51:28.8119849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8119932Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8120220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8120315Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8120692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8120782Z self_outputs = self.self( 2025-12-04T09:51:28.8121078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.8121251Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.8121575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.8121655Z x = self.pointwise(x) 2025-12-04T09:51:28.8121660Z 2025-12-04T09:51:28.8121779Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8122008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8122081Z res = mod(**inputs) 2025-12-04T09:51:28.8122398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8122489Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8122800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8122910Z hidden_states = self.encoder( 2025-12-04T09:51:28.8123224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8123309Z layer_outputs = layer_module( 2025-12-04T09:51:28.8123547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8123629Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8123923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8124010Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8124296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8124377Z self_outputs = self.self( 2025-12-04T09:51:28.8124687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.8124865Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.8124869Z 2025-12-04T09:51:28.8124979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8125193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8125281Z res = mod(**inputs) 2025-12-04T09:51:28.8125560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8125665Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8125949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8126026Z hidden_states = self.encoder( 2025-12-04T09:51:28.8126327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8126405Z layer_outputs = layer_module( 2025-12-04T09:51:28.8126644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8126733Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8127021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8127115Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8127403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8127477Z self_outputs = self.self( 2025-12-04T09:51:28.8127773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.8127906Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.8127912Z 2025-12-04T09:51:28.8128029Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8128241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8128312Z res = mod(**inputs) 2025-12-04T09:51:28.8128609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8128697Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8128991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8129078Z hidden_states = self.encoder( 2025-12-04T09:51:28.8129368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8129453Z layer_outputs = layer_module( 2025-12-04T09:51:28.8129772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8129857Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8130152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8130240Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8130534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8130612Z self_outputs = self.self( 2025-12-04T09:51:28.8130899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.8131045Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.8131048Z 2025-12-04T09:51:28.8131153Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8131267Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8131379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8131603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8131676Z res = mod(**inputs) 2025-12-04T09:51:28.8131966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8132064Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8132374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8132450Z hidden_states = self.encoder( 2025-12-04T09:51:28.8132746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8132825Z layer_outputs = layer_module( 2025-12-04T09:51:28.8133078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8133162Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8133450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8133543Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8133836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8133921Z self_outputs = self.self( 2025-12-04T09:51:28.8134211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.8134335Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.8134339Z 2025-12-04T09:51:28.8134461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8134675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8134745Z res = mod(**inputs) 2025-12-04T09:51:28.8135042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8135128Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8135437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8135517Z hidden_states = self.encoder( 2025-12-04T09:51:28.8135807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8135892Z layer_outputs = layer_module( 2025-12-04T09:51:28.8136168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8136279Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8136565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8136653Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8136948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.8137089Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.8137377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.8137476Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8137480Z 2025-12-04T09:51:28.8137588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8137830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8137904Z res = mod(**inputs) 2025-12-04T09:51:28.8138190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8138282Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8138569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8138652Z hidden_states = self.encoder( 2025-12-04T09:51:28.8138961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8139037Z layer_outputs = layer_module( 2025-12-04T09:51:28.8139285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8139368Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8139658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8139756Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8140038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8140129Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8140455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.8140589Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.8140884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.8140970Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8140973Z 2025-12-04T09:51:28.8141097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8141309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8141381Z res = mod(**inputs) 2025-12-04T09:51:28.8141674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8141761Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8142053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8142132Z hidden_states = self.encoder( 2025-12-04T09:51:28.8142421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8142505Z layer_outputs = layer_module( 2025-12-04T09:51:28.8142771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8142871Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8143168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8143258Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8143548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8143632Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8143959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.8144099Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.8144405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.8144539Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.8144776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.8144854Z return self.act(input) 2025-12-04T09:51:28.8144858Z 2025-12-04T09:51:28.8144975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8145190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8145261Z res = mod(**inputs) 2025-12-04T09:51:28.8145556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8145663Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8145965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8146045Z hidden_states = self.encoder( 2025-12-04T09:51:28.8146333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8146418Z layer_outputs = layer_module( 2025-12-04T09:51:28.8146657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8146739Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8147032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8147269Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8147574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8147657Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8148000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.8148163Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.8148462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.8148559Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8148563Z 2025-12-04T09:51:28.8148676Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8148895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8148977Z res = mod(**inputs) 2025-12-04T09:51:28.8149277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8149373Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8149748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8149851Z hidden_states = self.encoder( 2025-12-04T09:51:28.8150150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8150229Z layer_outputs = layer_module( 2025-12-04T09:51:28.8150471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8150567Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8150856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8150953Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8151257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8151362Z self_outputs = self.self( 2025-12-04T09:51:28.8151661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.8151760Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.8151764Z 2025-12-04T09:51:28.8151886Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8152102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8152174Z res = mod(**inputs) 2025-12-04T09:51:28.8152476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8152593Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8152882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8152968Z hidden_states = self.encoder( 2025-12-04T09:51:28.8153258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8153345Z layer_outputs = layer_module( 2025-12-04T09:51:28.8153585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8153668Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8153962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8154050Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8154336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8154418Z self_outputs = self.self( 2025-12-04T09:51:28.8154708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.8154803Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.8154806Z 2025-12-04T09:51:28.8154916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8155127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8155203Z res = mod(**inputs) 2025-12-04T09:51:28.8155494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8155587Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8155874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8155951Z hidden_states = self.encoder( 2025-12-04T09:51:28.8156242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8156355Z layer_outputs = layer_module( 2025-12-04T09:51:28.8156593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8156684Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8156971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8157064Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8157349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8157426Z self_outputs = self.self( 2025-12-04T09:51:28.8157719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.8157817Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.8157841Z 2025-12-04T09:51:28.8157937Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8158024Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8158133Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8158348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8158418Z res = mod(**inputs) 2025-12-04T09:51:28.8158705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8158798Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8159115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8159198Z hidden_states = self.encoder( 2025-12-04T09:51:28.8159490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8159570Z layer_outputs = layer_module( 2025-12-04T09:51:28.8159824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8159910Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8160202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8160302Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8160655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8160751Z self_outputs = self.self( 2025-12-04T09:51:28.8161047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.8161165Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.8161170Z 2025-12-04T09:51:28.8161268Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8161383Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8161611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8161685Z res = mod(**inputs) 2025-12-04T09:51:28.8161981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8162081Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8162374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8162456Z hidden_states = self.encoder( 2025-12-04T09:51:28.8162760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8162847Z layer_outputs = layer_module( 2025-12-04T09:51:28.8163109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8163204Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8163478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8163568Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8163842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8163923Z self_outputs = self.self( 2025-12-04T09:51:28.8164196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.8164357Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.8164668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.8164755Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.8164759Z 2025-12-04T09:51:28.8164878Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8165091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8165160Z res = mod(**inputs) 2025-12-04T09:51:28.8165460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8165568Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8165854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8165939Z hidden_states = self.encoder( 2025-12-04T09:51:28.8166229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8166315Z layer_outputs = layer_module( 2025-12-04T09:51:28.8166560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8166637Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8166914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8166996Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8167280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8167364Z self_outputs = self.self( 2025-12-04T09:51:28.8167651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.8167831Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.8168121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.8168195Z x = self.pointwise(x) 2025-12-04T09:51:28.8168199Z 2025-12-04T09:51:28.8168317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8168529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8168607Z res = mod(**inputs) 2025-12-04T09:51:28.8168891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8168979Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8169274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8169353Z hidden_states = self.encoder( 2025-12-04T09:51:28.8169659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8169764Z layer_outputs = layer_module( 2025-12-04T09:51:28.8169999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8170088Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8170375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8170462Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8170756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8170831Z self_outputs = self.self( 2025-12-04T09:51:28.8171141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.8171309Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.8171313Z 2025-12-04T09:51:28.8171423Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8171644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8171714Z res = mod(**inputs) 2025-12-04T09:51:28.8172004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8172116Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8172407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8172490Z hidden_states = self.encoder( 2025-12-04T09:51:28.8172780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8172857Z layer_outputs = layer_module( 2025-12-04T09:51:28.8173101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8173180Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8173470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8173554Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8173840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8173925Z self_outputs = self.self( 2025-12-04T09:51:28.8174208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.8174340Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.8174354Z 2025-12-04T09:51:28.8174463Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8174676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8174754Z res = mod(**inputs) 2025-12-04T09:51:28.8175039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8175124Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8175418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8175497Z hidden_states = self.encoder( 2025-12-04T09:51:28.8175802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8175876Z layer_outputs = layer_module( 2025-12-04T09:51:28.8176125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8176229Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8176498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8176581Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8176856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8176928Z self_outputs = self.self( 2025-12-04T09:51:28.8177203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.8177331Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.8177335Z 2025-12-04T09:51:28.8177415Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8177526Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8177635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8177846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8177914Z res = mod(**inputs) 2025-12-04T09:51:28.8178194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8178284Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8178556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8178648Z hidden_states = self.encoder( 2025-12-04T09:51:28.8178927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8178998Z layer_outputs = layer_module( 2025-12-04T09:51:28.8179231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8179312Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8179586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8179677Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8179948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8180021Z self_outputs = self.self( 2025-12-04T09:51:28.8180316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.8180436Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.8180440Z 2025-12-04T09:51:28.8180560Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8180773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8180845Z res = mod(**inputs) 2025-12-04T09:51:28.8181139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8181225Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8181522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8181599Z hidden_states = self.encoder( 2025-12-04T09:51:28.8181888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8181980Z layer_outputs = layer_module( 2025-12-04T09:51:28.8182204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8182300Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8182600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8182679Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8182959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.8183089Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.8183366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.8183460Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8183464Z 2025-12-04T09:51:28.8183568Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8183804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8183876Z res = mod(**inputs) 2025-12-04T09:51:28.8184149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8184238Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8184509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8184590Z hidden_states = self.encoder( 2025-12-04T09:51:28.8184863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8184956Z layer_outputs = layer_module( 2025-12-04T09:51:28.8185187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8185265Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8185539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8185636Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8185900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8185988Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8186289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.8186415Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.8186696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.8186780Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8186783Z 2025-12-04T09:51:28.8186893Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8187093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8187159Z res = mod(**inputs) 2025-12-04T09:51:28.8187437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8187518Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8187789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8187871Z hidden_states = self.encoder( 2025-12-04T09:51:28.8188148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8188228Z layer_outputs = layer_module( 2025-12-04T09:51:28.8188467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8188574Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8188885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8188973Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8189258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8189341Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8189661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.8189800Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.8190088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.8190226Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.8190473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.8190549Z return self.act(input) 2025-12-04T09:51:28.8190552Z 2025-12-04T09:51:28.8190668Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8190880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8190951Z res = mod(**inputs) 2025-12-04T09:51:28.8191246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8191355Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8191648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8191724Z hidden_states = self.encoder( 2025-12-04T09:51:28.8192013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8192097Z layer_outputs = layer_module( 2025-12-04T09:51:28.8192336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8192418Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8192713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8192801Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8193090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8193170Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8193493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.8193651Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.8193937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.8194032Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8194036Z 2025-12-04T09:51:28.8194145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8194357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8194436Z res = mod(**inputs) 2025-12-04T09:51:28.8194721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8194807Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8195103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8195221Z hidden_states = self.encoder( 2025-12-04T09:51:28.8195522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8195598Z layer_outputs = layer_module( 2025-12-04T09:51:28.8195837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8195926Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8196214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8196310Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8196599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8196675Z self_outputs = self.self( 2025-12-04T09:51:28.8196991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:51:28.8197095Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:51:28.8197099Z 2025-12-04T09:51:28.8197208Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8197429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8197506Z res = mod(**inputs) 2025-12-04T09:51:28.8197803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8197914Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8198199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8198283Z hidden_states = self.encoder( 2025-12-04T09:51:28.8198576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8198660Z layer_outputs = layer_module( 2025-12-04T09:51:28.8198898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8198983Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8199278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8199367Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8199657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8199741Z self_outputs = self.self( 2025-12-04T09:51:28.8200030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:51:28.8200128Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:51:28.8200133Z 2025-12-04T09:51:28.8200246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8200460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8200616Z res = mod(**inputs) 2025-12-04T09:51:28.8200935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8201025Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8201346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8201425Z hidden_states = self.encoder( 2025-12-04T09:51:28.8201735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8201816Z layer_outputs = layer_module( 2025-12-04T09:51:28.8202115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8202210Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8202500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8202594Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8202884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8202962Z self_outputs = self.self( 2025-12-04T09:51:28.8203259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:51:28.8203358Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:51:28.8203363Z 2025-12-04T09:51:28.8203476Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8203569Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8203682Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8203900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8203970Z res = mod(**inputs) 2025-12-04T09:51:28.8204258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8204353Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8204638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8204739Z hidden_states = self.encoder( 2025-12-04T09:51:28.8205027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8205103Z layer_outputs = layer_module( 2025-12-04T09:51:28.8205350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8205434Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8205722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8205818Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8206106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8206188Z self_outputs = self.self( 2025-12-04T09:51:28.8206475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:51:28.8206584Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:51:28.8206588Z 2025-12-04T09:51:28.8206679Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8206791Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8207003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8207080Z res = mod(**inputs) 2025-12-04T09:51:28.8207369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8207463Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8207751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8207828Z hidden_states = self.encoder( 2025-12-04T09:51:28.8208124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8208200Z layer_outputs = layer_module( 2025-12-04T09:51:28.8208461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8208571Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8208859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8208956Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8209244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8209319Z self_outputs = self.self( 2025-12-04T09:51:28.8209618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.8209789Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.8210097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:51:28.8210187Z x = self.depthwise(hidden_states) 2025-12-04T09:51:28.8210191Z 2025-12-04T09:51:28.8210303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8210526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8210601Z res = mod(**inputs) 2025-12-04T09:51:28.8210918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8211010Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8211332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8211421Z hidden_states = self.encoder( 2025-12-04T09:51:28.8211731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8211811Z layer_outputs = layer_module( 2025-12-04T09:51:28.8212067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8212153Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8212460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8212551Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8212855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8212943Z self_outputs = self.self( 2025-12-04T09:51:28.8213238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:51:28.8213418Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:51:28.8213730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:51:28.8213807Z x = self.pointwise(x) 2025-12-04T09:51:28.8213811Z 2025-12-04T09:51:28.8213932Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8214151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8214223Z res = mod(**inputs) 2025-12-04T09:51:28.8214592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8214685Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8214992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8215069Z hidden_states = self.encoder( 2025-12-04T09:51:28.8215408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8215512Z layer_outputs = layer_module( 2025-12-04T09:51:28.8215762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8215857Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8216172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8216260Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8216575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8216652Z self_outputs = self.self( 2025-12-04T09:51:28.8216969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:51:28.8217168Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:51:28.8217173Z 2025-12-04T09:51:28.8217288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8217514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8217587Z res = mod(**inputs) 2025-12-04T09:51:28.8217896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8217992Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8218311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8218398Z hidden_states = self.encoder( 2025-12-04T09:51:28.8218717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8218800Z layer_outputs = layer_module( 2025-12-04T09:51:28.8219061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8219147Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8219455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8219554Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8219865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8219950Z self_outputs = self.self( 2025-12-04T09:51:28.8220245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:51:28.8220377Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:51:28.8220381Z 2025-12-04T09:51:28.8220504Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8220724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8220801Z res = mod(**inputs) 2025-12-04T09:51:28.8221100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8221188Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8221494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8221577Z hidden_states = self.encoder( 2025-12-04T09:51:28.8221877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8221962Z layer_outputs = layer_module( 2025-12-04T09:51:28.8222232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8222342Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8222640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8222729Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8223032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8223111Z self_outputs = self.self( 2025-12-04T09:51:28.8223414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:51:28.8223554Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:51:28.8223558Z 2025-12-04T09:51:28.8223648Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8223762Z cudagraph partition due to non gpu ops 2025-12-04T09:51:28.8223880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8224098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8224176Z res = mod(**inputs) 2025-12-04T09:51:28.8224474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8224570Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8224869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8224972Z hidden_states = self.encoder( 2025-12-04T09:51:28.8225296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8225374Z layer_outputs = layer_module( 2025-12-04T09:51:28.8225630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8225718Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8226017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8226113Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8226416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:51:28.8226491Z self_outputs = self.self( 2025-12-04T09:51:28.8226790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:51:28.8226913Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:51:28.8226917Z 2025-12-04T09:51:28.8227035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8227248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8227319Z res = mod(**inputs) 2025-12-04T09:51:28.8227618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8227703Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8228001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8228078Z hidden_states = self.encoder( 2025-12-04T09:51:28.8228364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8228450Z layer_outputs = layer_module( 2025-12-04T09:51:28.8228692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8228776Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8229103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:51:28.8229210Z self_attention_outputs = self.attention( 2025-12-04T09:51:28.8229507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:51:28.8229649Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:51:28.8229941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:51:28.8230039Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8230042Z 2025-12-04T09:51:28.8230152Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8230370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8230462Z res = mod(**inputs) 2025-12-04T09:51:28.8230753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8230850Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8231134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8231212Z hidden_states = self.encoder( 2025-12-04T09:51:28.8231508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8231640Z layer_outputs = layer_module( 2025-12-04T09:51:28.8231883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8231966Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8232260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8232359Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8232640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8232728Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8233052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.8233181Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.8233479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:51:28.8233566Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8233570Z 2025-12-04T09:51:28.8233677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8233900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8233974Z res = mod(**inputs) 2025-12-04T09:51:28.8234266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8234354Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8234639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8234728Z hidden_states = self.encoder( 2025-12-04T09:51:28.8235026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8235107Z layer_outputs = layer_module( 2025-12-04T09:51:28.8235332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8235411Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8235731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8235818Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8236084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8236173Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8236479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:51:28.8236613Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:51:28.8236885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:51:28.8237004Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:51:28.8237276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:51:28.8237354Z return self.act(input) 2025-12-04T09:51:28.8237359Z 2025-12-04T09:51:28.8237472Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8237678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8237744Z res = mod(**inputs) 2025-12-04T09:51:28.8238024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:51:28.8238126Z generator_hidden_states = self.convbert( 2025-12-04T09:51:28.8238392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:51:28.8238472Z hidden_states = self.encoder( 2025-12-04T09:51:28.8238744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:51:28.8238823Z layer_outputs = layer_module( 2025-12-04T09:51:28.8239048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:51:28.8239126Z return super().__call__(*args, **kwargs) 2025-12-04T09:51:28.8239401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:51:28.8239484Z layer_output = apply_chunking_to_forward( 2025-12-04T09:51:28.8239755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:51:28.8239834Z return forward_fn(*input_tensors) 2025-12-04T09:51:28.8240139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:51:28.8240283Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:51:28.8240631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:51:28.8240732Z hidden_states = self.dense(hidden_states) 2025-12-04T09:51:28.8240744Z 2025-12-04T09:51:28.8240859Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8241080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8241161Z res = mod(**inputs) 2025-12-04T09:51:28.8241462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 937, in forward 2025-12-04T09:51:28.8241635Z prediction_scores = self.generator_predictions(generator_sequence_output) 2025-12-04T09:51:28.8241950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 875, in forward 2025-12-04T09:51:28.8242091Z hidden_states = self.dense(generator_hidden_states) 2025-12-04T09:51:28.8242112Z 2025-12-04T09:51:28.8242298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8242512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8242582Z res = mod(**inputs) 2025-12-04T09:51:28.8242875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 938, in forward 2025-12-04T09:51:28.8243005Z prediction_scores = self.generator_lm_head(prediction_scores) 2025-12-04T09:51:28.8243011Z 2025-12-04T09:51:28.8243120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:51:28.8243320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:51:28.8243387Z res = mod(**inputs) 2025-12-04T09:51:28.8243690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 944, in forward 2025-12-04T09:51:28.8243870Z loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:51:28.8243874Z 2025-12-04T09:51:41.0469779Z Compilation time (from dynamo_timed): 24.037567074 2025-12-04T09:51:41.0505425Z pass 2025-12-04T09:51:41.0505838Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:51:41.0506688Z TIMING: _recursive_pre_grad_passes:0.01033 _recursive_joint_graph_passes:0.65255 _recursive_post_grad_passes:0.51714 async_compile.wait:0.75191 code_gen:11.44198 inductor_compile:14.09263 backend_compile:19.32207 gc:0.00029 entire_frame_compile:24.03757 total_wall_time:24.03757 2025-12-04T09:51:41.0508016Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:13628 | FakeTensor.__torch_dispatch__:7175 | ProxyTorchDispatchMode.__torch_dispatch__:3966 2025-12-04T09:51:41.0508566Z Dynamo produced 1 graphs covering 634 ops with 0 graph breaks (0 unique) 2025-12-04T09:51:42.9227960Z accuracy pass_rate=92.59% 2025-12-04T09:51:42.9234577Z calls_captured gmean=0.00x mean=579.519x 2025-12-04T09:51:42.9238345Z unique_graphs gmean=0.00x mean=1.111x 2025-12-04T09:51:42.9238712Z graph_breaks gmean=0.00x mean=0.222x 2025-12-04T09:51:42.9238982Z unique_graph_breaks gmean=0.00x mean=0.074x 2025-12-04T09:51:42.9239241Z autograd_captures gmean=0.00x mean=0.000x 2025-12-04T09:51:42.9240165Z autograd_compiles gmean=0.00x mean=0.000x 2025-12-04T09:51:42.9244059Z cudagraph_skips gmean=0.00x mean=1.111x 2025-12-04T09:51:42.9244671Z compilation_latency mean=21.825 seconds 2025-12-04T09:51:44.0714864Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-12-04T09:51:44.3981554Z AlbertForMaskedLM PASS 2025-12-04T09:51:44.3981907Z AllenaiLongformerBase PASS 2025-12-04T09:51:44.3984753Z BartForCausalLM PASS 2025-12-04T09:51:44.3990146Z BertForMaskedLM PASS 2025-12-04T09:51:44.3996523Z BlenderbotForCausalLM XFAIL 2025-12-04T09:51:44.3996847Z DebertaV2ForMaskedLM XFAIL 2025-12-04T09:51:44.4000257Z DistilBertForMaskedLM PASS 2025-12-04T09:51:44.4005036Z DistillGPT2 PASS 2025-12-04T09:51:44.4008410Z ElectraForCausalLM PASS 2025-12-04T09:51:44.4010226Z GPT2ForSequenceClassification PASS 2025-12-04T09:51:44.4012968Z GoogleFnet PASS 2025-12-04T09:51:44.4016880Z LayoutLMForMaskedLM PASS 2025-12-04T09:51:44.4021064Z M2M100ForConditionalGeneration PASS 2025-12-04T09:51:44.4026383Z MBartForCausalLM PASS 2025-12-04T09:51:44.4031163Z MT5ForConditionalGeneration PASS 2025-12-04T09:51:44.4031909Z MegatronBertForCausalLM PASS 2025-12-04T09:51:44.4038600Z MobileBertForMaskedLM PASS 2025-12-04T09:51:44.4038953Z OPTForCausalLM PASS 2025-12-04T09:51:44.4045663Z PLBartForCausalLM PASS 2025-12-04T09:51:44.4049608Z PegasusForCausalLM PASS 2025-12-04T09:51:44.4060942Z RobertaForCausalLM PASS 2025-12-04T09:51:44.4065793Z T5ForConditionalGeneration PASS 2025-12-04T09:51:44.4070333Z T5Small PASS 2025-12-04T09:51:44.4072847Z TrOCRForCausalLM PASS 2025-12-04T09:51:44.4077315Z XGLMForCausalLM PASS 2025-12-04T09:51:44.4079968Z XLNetLMHeadModel PASS 2025-12-04T09:51:44.4080623Z YituTechConvBert PASS 2025-12-04T09:51:44.4653833Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-12-04T09:51:44.7832212Z AlbertForMaskedLM PASS 2025-12-04T09:51:44.7833490Z AllenaiLongformerBase PASS 2025-12-04T09:51:44.7836269Z BartForCausalLM PASS 2025-12-04T09:51:44.7839272Z BertForMaskedLM PASS 2025-12-04T09:51:44.7842761Z BlenderbotForCausalLM PASS 2025-12-04T09:51:44.7847851Z DebertaV2ForMaskedLM PASS 2025-12-04T09:51:44.7853685Z DistilBertForMaskedLM PASS 2025-12-04T09:51:44.7865174Z DistillGPT2 PASS 2025-12-04T09:51:44.7869754Z ElectraForCausalLM PASS 2025-12-04T09:51:44.7874076Z GPT2ForSequenceClassification PASS 2025-12-04T09:51:44.7874741Z GoogleFnet PASS 2025-12-04T09:51:44.7876677Z LayoutLMForMaskedLM PASS 2025-12-04T09:51:44.7881247Z M2M100ForConditionalGeneration PASS 2025-12-04T09:51:44.7891813Z MBartForCausalLM PASS 2025-12-04T09:51:44.7892239Z MT5ForConditionalGeneration PASS 2025-12-04T09:51:44.7897687Z MegatronBertForCausalLM PASS 2025-12-04T09:51:44.7900044Z MobileBertForMaskedLM PASS 2025-12-04T09:51:44.7909494Z OPTForCausalLM PASS 2025-12-04T09:51:44.7915886Z PLBartForCausalLM PASS 2025-12-04T09:51:44.7916349Z PegasusForCausalLM PASS 2025-12-04T09:51:44.7916623Z RobertaForCausalLM PASS 2025-12-04T09:51:44.7916873Z T5ForConditionalGeneration PASS 2025-12-04T09:51:44.7917457Z T5Small PASS 2025-12-04T09:51:44.7929457Z TrOCRForCausalLM PASS 2025-12-04T09:51:44.7930487Z XGLMForCausalLM PASS_BUT_FLAKY 2025-12-04T09:51:44.7935271Z XLNetLMHeadModel PASS 2025-12-04T09:51:44.7941287Z YituTechConvBert PASS 2025-12-04T09:51:44.8512199Z + sccache_epilogue 2025-12-04T09:51:44.8512624Z + echo '::group::Sccache Compilation Log' 2025-12-04T09:51:44.8513735Z ##[group]Sccache Compilation Log 2025-12-04T09:51:44.8514033Z + echo '=================== sccache compilation log ===================' 2025-12-04T09:51:44.8514325Z =================== sccache compilation log =================== 2025-12-04T09:51:44.8514754Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-12-04T09:51:44.8768612Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-12-04T09:51:44.8769126Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-12-04T09:51:44.8769475Z + sccache --show-stats 2025-12-04T09:51:44.8799329Z Compile requests 276 2025-12-04T09:51:44.8799614Z Compile requests executed 0 2025-12-04T09:51:44.8799845Z Cache hits 0 2025-12-04T09:51:44.8800097Z Cache misses 0 2025-12-04T09:51:44.8800359Z Cache hits rate - 2025-12-04T09:51:44.8800976Z Cache timeouts 0 2025-12-04T09:51:44.8801286Z Cache read errors 0 2025-12-04T09:51:44.8801543Z Forced recaches 0 2025-12-04T09:51:44.8801768Z Cache write errors 0 2025-12-04T09:51:44.8801988Z Cache errors 0 2025-12-04T09:51:44.8802210Z Compilations 0 2025-12-04T09:51:44.8802441Z Compilation failures 0 2025-12-04T09:51:44.8802674Z Non-cacheable compilations 0 2025-12-04T09:51:44.8802902Z Non-cacheable calls 25 2025-12-04T09:51:44.8803141Z Non-compilation calls 251 2025-12-04T09:51:44.8803370Z Unsupported compiler calls 0 2025-12-04T09:51:44.8803598Z Average cache write 0.000 s 2025-12-04T09:51:44.8803833Z Average compiler 0.000 s 2025-12-04T09:51:44.8804068Z Average cache read hit 0.000 s 2025-12-04T09:51:44.8804348Z Failed distributed compilations 0 2025-12-04T09:51:44.8804521Z 2025-12-04T09:51:44.8804602Z Non-cacheable reasons: 2025-12-04T09:51:44.8804804Z -E 25 2025-12-04T09:51:44.8804954Z 2025-12-04T09:51:44.8805130Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-12-04T09:51:44.8805449Z Version (client) 0.10.0 2025-12-04T09:51:44.8805684Z + sccache --stop-server 2025-12-04T09:51:44.8825201Z Stopping sccache server... 2025-12-04T09:51:44.8829145Z Compile requests 276 2025-12-04T09:51:44.8829488Z Compile requests executed 0 2025-12-04T09:51:44.8830615Z Cache hits 0 2025-12-04T09:51:44.8831296Z Cache misses 0 2025-12-04T09:51:44.8831990Z Cache hits rate - 2025-12-04T09:51:44.8832306Z Cache timeouts 0 2025-12-04T09:51:44.8832573Z Cache read errors 0 2025-12-04T09:51:44.8832807Z Forced recaches 0 2025-12-04T09:51:44.8833051Z Cache write errors 0 2025-12-04T09:51:44.8833269Z Cache errors 0 2025-12-04T09:51:44.8833481Z Compilations 0 2025-12-04T09:51:44.8833705Z Compilation failures 0 2025-12-04T09:51:44.8833935Z Non-cacheable compilations 0 2025-12-04T09:51:44.8834159Z Non-cacheable calls 25 2025-12-04T09:51:44.8834374Z Non-compilation calls 251 2025-12-04T09:51:44.8834600Z Unsupported compiler calls 0 2025-12-04T09:51:44.8834835Z Average cache write 0.000 s 2025-12-04T09:51:44.8835071Z Average compiler 0.000 s 2025-12-04T09:51:44.8835303Z Average cache read hit 0.000 s 2025-12-04T09:51:44.8835537Z Failed distributed compilations 0 2025-12-04T09:51:44.8835688Z 2025-12-04T09:51:44.8835769Z Non-cacheable reasons: 2025-12-04T09:51:44.8835975Z -E 25 2025-12-04T09:51:44.8836123Z 2025-12-04T09:51:44.8836307Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-12-04T09:51:44.8836630Z Version (client) 0.10.0 2025-12-04T09:51:44.8836903Z + echo ::endgroup:: 2025-12-04T09:51:44.8837510Z ##[endgroup] 2025-12-04T09:51:44.8837689Z + cleanup_workspace 2025-12-04T09:51:44.8838040Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-12-04T09:51:44.8838570Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-12-04T09:51:44.8838997Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-12-04T09:51:44.8839346Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-12-04T09:51:44.8839724Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-12-04T09:51:44.8840115Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-12-04T09:51:44.8840933Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-12-04T09:51:45.3504470Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-12-04T09:51:45.3504856Z with: 2025-12-04T09:51:45.3505092Z benchmark-results-dir: test/test-reports 2025-12-04T09:51:45.3505358Z dry-run: false 2025-12-04T09:51:45.3505569Z schema-version: v3 2025-12-04T09:51:45.3506027Z github-token: *** 2025-12-04T09:51:45.3506248Z env: 2025-12-04T09:51:45.3506436Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:45.3506672Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:45.3507048Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:45.3507446Z ##[endgroup] 2025-12-04T09:51:45.3521949Z ##[group]Run set -eux 2025-12-04T09:51:45.3522198Z set -eux 2025-12-04T09:51:45.3522372Z  2025-12-04T09:51:45.3522539Z if [[ -n "" ]]; then 2025-12-04T09:51:45.3522743Z  source "" 2025-12-04T09:51:45.3522926Z fi 2025-12-04T09:51:45.3523220Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-12-04T09:51:45.3523509Z  2025-12-04T09:51:45.3523675Z DEVICE_NAME="" 2025-12-04T09:51:45.3523870Z DEVICE_TYPE="" 2025-12-04T09:51:45.3524055Z  2025-12-04T09:51:45.3524229Z if command -v nvidia-smi; then 2025-12-04T09:51:45.3524550Z  # NB: I'm using PyTorch here to get the device name, however, it needs to 2025-12-04T09:51:45.3524939Z  # install the correct version of PyTorch manually for now. Any PyTorch 2025-12-04T09:51:45.3525384Z  # version is fine, I just use 2.7.1 to satify PYPIDEP linter 2025-12-04T09:51:45.3525688Z  python3 -mpip install torch==2.7.1 2025-12-04T09:51:45.3525943Z elif command -v rocminfo; then 2025-12-04T09:51:45.3526253Z  # NB: Installing torch on ROCm runner with pip here causes CI to fail 2025-12-04T09:51:45.3526627Z  # with a memoryview is too large error only on MI300 runners. Is pip 2025-12-04T09:51:45.3527015Z  # version on ROCm runner there too old? As a workaround, let's use the 2025-12-04T09:51:45.3527351Z  # GPU device name coming from rocminfo instead 2025-12-04T09:51:45.3527598Z  DEVICE_NAME=rocm 2025-12-04T09:51:45.3527936Z  DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs) 2025-12-04T09:51:45.3528272Z fi 2025-12-04T09:51:45.3528437Z  2025-12-04T09:51:45.3528636Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV 2025-12-04T09:51:45.3528925Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV 2025-12-04T09:51:45.3537718Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:45.3537980Z env: 2025-12-04T09:51:45.3538163Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:45.3538368Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:45.3538696Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:45.3539055Z ##[endgroup] 2025-12-04T09:51:45.3571873Z + [[ -n '' ]] 2025-12-04T09:51:45.3572229Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-12-04T09:51:45.5518481Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T09:51:46.4008177Z Collecting boto3==1.35.33 2025-12-04T09:51:46.4144775Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2025-12-04T09:51:46.6647331Z Collecting psutil==7.0.0 2025-12-04T09:51:46.6678602Z Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (277 kB) 2025-12-04T09:51:46.6950377Z Collecting pynvml==12.0.0 2025-12-04T09:51:46.6981886Z Downloading pynvml-12.0.0-py3-none-any.whl (26 kB) 2025-12-04T09:51:46.7065023Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2025-12-04T09:51:47.6252144Z Collecting botocore<1.36.0,>=1.35.33 2025-12-04T09:51:47.6291230Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-12-04T09:51:47.7546129Z Collecting s3transfer<0.11.0,>=0.10.0 2025-12-04T09:51:47.7577732Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-12-04T09:51:47.7993802Z Collecting nvidia-ml-py<13.0.0a0,>=12.0.0 2025-12-04T09:51:47.8025620Z Downloading nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB) 2025-12-04T09:51:47.8105511Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2025-12-04T09:51:47.8112712Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2025-12-04T09:51:47.9296626Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2025-12-04T09:51:48.0492372Z Installing collected packages: botocore, s3transfer, nvidia-ml-py, pynvml, psutil, boto3 2025-12-04T09:51:48.4255929Z Attempting uninstall: nvidia-ml-py 2025-12-04T09:51:48.4260241Z Found existing installation: nvidia-ml-py 11.525.84 2025-12-04T09:51:48.4265003Z Uninstalling nvidia-ml-py-11.525.84: 2025-12-04T09:51:48.4413836Z Successfully uninstalled nvidia-ml-py-11.525.84 2025-12-04T09:51:48.4943789Z Attempting uninstall: psutil 2025-12-04T09:51:48.4947392Z Found existing installation: psutil 5.9.8 2025-12-04T09:51:48.4991419Z Uninstalling psutil-5.9.8: 2025-12-04T09:51:48.4997178Z Successfully uninstalled psutil-5.9.8 2025-12-04T09:51:48.6441748Z Successfully installed boto3-1.35.33 botocore-1.35.99 nvidia-ml-py-12.575.51 psutil-7.0.0 pynvml-12.0.0 s3transfer-0.10.4 2025-12-04T09:51:48.7709522Z + DEVICE_NAME= 2025-12-04T09:51:48.7709795Z + DEVICE_TYPE= 2025-12-04T09:51:48.7710068Z + command -v nvidia-smi 2025-12-04T09:51:48.7710308Z + command -v rocminfo 2025-12-04T09:51:48.7710495Z + echo DEVICE_NAME= 2025-12-04T09:51:48.7710678Z + echo DEVICE_TYPE= 2025-12-04T09:51:48.7731580Z ##[group]Run set -eux 2025-12-04T09:51:48.7731796Z set -eux 2025-12-04T09:51:48.7731973Z  2025-12-04T09:51:48.7732164Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-12-04T09:51:48.7732420Z  echo "Missing github-token input" 2025-12-04T09:51:48.7732652Z  exit 1 2025-12-04T09:51:48.7732826Z fi 2025-12-04T09:51:48.7737670Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:48.7737935Z env: 2025-12-04T09:51:48.7738268Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:48.7738481Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:48.7738819Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:48.7739245Z DEVICE_NAME: 2025-12-04T09:51:48.7739419Z DEVICE_TYPE: 2025-12-04T09:51:48.7739824Z GITHUB_TOKEN: *** 2025-12-04T09:51:48.7740006Z ##[endgroup] 2025-12-04T09:51:48.7762915Z + [[ -z *** ]] 2025-12-04T09:51:48.7795320Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-12-04T09:51:48.7795675Z with: 2025-12-04T09:51:48.7796036Z github-token: *** 2025-12-04T09:51:48.7796229Z env: 2025-12-04T09:51:48.7796426Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:48.7796668Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:48.7797044Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:48.7797451Z DEVICE_NAME: 2025-12-04T09:51:48.7797656Z DEVICE_TYPE: 2025-12-04T09:51:48.7797847Z ##[endgroup] 2025-12-04T09:51:48.7810836Z ##[group]Run set -eux 2025-12-04T09:51:48.7811090Z set -eux 2025-12-04T09:51:48.7811290Z  2025-12-04T09:51:48.7811671Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-12-04T09:51:48.7817044Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:48.7817349Z env: 2025-12-04T09:51:48.7817559Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:48.7817797Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:48.7818367Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:48.7818764Z DEVICE_NAME: 2025-12-04T09:51:48.7818970Z DEVICE_TYPE: 2025-12-04T09:51:48.7819332Z GITHUB_TOKEN: *** 2025-12-04T09:51:48.7819539Z ##[endgroup] 2025-12-04T09:51:48.7841614Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 19923066595 i-0c5ea43668efc70fe 2025-12-04T09:51:50.9103478Z setting job-id=57118563290 2025-12-04T09:51:50.9104078Z setting job-name=periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:51:50.9208414Z ##[group]Run set -eux 2025-12-04T09:51:50.9208640Z set -eux 2025-12-04T09:51:50.9208803Z  2025-12-04T09:51:50.9208974Z if [[ -n "" ]]; then 2025-12-04T09:51:50.9209174Z  source "" 2025-12-04T09:51:50.9209340Z fi 2025-12-04T09:51:50.9209518Z  2025-12-04T09:51:50.9209788Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-12-04T09:51:50.9210131Z  --schema-version "${SCHEMA_VERSION}" \ 2025-12-04T09:51:50.9210367Z  --repo "${REPO}" \ 2025-12-04T09:51:50.9210581Z  --head-branch "${HEAD_BRANCH}" \ 2025-12-04T09:51:50.9210814Z  --head-sha "${HEAD_SHA}" \ 2025-12-04T09:51:50.9211045Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-12-04T09:51:50.9211290Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-12-04T09:51:50.9211607Z  --job-id "${JOB_ID}" \ 2025-12-04T09:51:50.9211816Z  --job-name "${JOB_NAME}" 2025-12-04T09:51:50.9216873Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:50.9217123Z env: 2025-12-04T09:51:50.9217297Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:50.9217480Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:50.9217801Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:50.9218142Z DEVICE_NAME: 2025-12-04T09:51:50.9218312Z DEVICE_TYPE: 2025-12-04T09:51:50.9218476Z SCHEMA_VERSION: v3 2025-12-04T09:51:50.9218656Z REPO: pytorch/pytorch 2025-12-04T09:51:50.9218833Z HEAD_BRANCH: refs/heads/main 2025-12-04T09:51:50.9219062Z HEAD_SHA: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:51:50.9219297Z WORKFLOW_RUN_ID: 19923066595 2025-12-04T09:51:50.9219475Z RUN_ATTEMPT: 1 2025-12-04T09:51:50.9219646Z JOB_ID: 57118563290 2025-12-04T09:51:50.9220027Z JOB_NAME: periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:51:50.9220430Z ##[endgroup] 2025-12-04T09:51:50.9242729Z + [[ -n '' ]] 2025-12-04T09:51:50.9244311Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 --workflow-id 19923066595 --run-attempt 1 --job-id 57118563290 --job-name 'periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)' 2025-12-04T09:51:50.9516656Z ##[group]Run set -eux 2025-12-04T09:51:50.9516860Z set -eux 2025-12-04T09:51:50.9517029Z  2025-12-04T09:51:50.9517193Z if [[ -n "" ]]; then 2025-12-04T09:51:50.9517392Z  source "" 2025-12-04T09:51:50.9517573Z fi 2025-12-04T09:51:50.9517728Z  2025-12-04T09:51:50.9518019Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py" 2025-12-04T09:51:50.9523167Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:50.9523426Z env: 2025-12-04T09:51:50.9523625Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:50.9523816Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:50.9524135Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:50.9524684Z DEVICE_NAME: 2025-12-04T09:51:50.9524858Z DEVICE_TYPE: 2025-12-04T09:51:50.9525035Z ##[endgroup] 2025-12-04T09:51:50.9546354Z + [[ -n '' ]] 2025-12-04T09:51:50.9547817Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py 2025-12-04T09:51:50.9886623Z INFO:root:Fail to import torch to get the device name 2025-12-04T09:51:50.9980822Z ##[group]Run set -eux 2025-12-04T09:51:50.9981025Z set -eux 2025-12-04T09:51:50.9981199Z  2025-12-04T09:51:50.9981381Z # TODO (huydhn): Implement this part 2025-12-04T09:51:50.9981632Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:51:50.9986350Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:50.9986609Z env: 2025-12-04T09:51:50.9986774Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:50.9986974Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:50.9987308Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:50.9987626Z DEVICE_NAME: 2025-12-04T09:51:50.9987782Z DEVICE_TYPE: 2025-12-04T09:51:50.9987947Z ##[endgroup] 2025-12-04T09:51:51.0010025Z + echo 'dependencies={}' 2025-12-04T09:51:51.0031332Z ##[group]Run set -eux 2025-12-04T09:51:51.0031555Z set -eux 2025-12-04T09:51:51.0031725Z  2025-12-04T09:51:51.0031894Z if [[ -n "" ]]; then 2025-12-04T09:51:51.0032087Z  source "" 2025-12-04T09:51:51.0032259Z fi 2025-12-04T09:51:51.0032496Z  2025-12-04T09:51:51.0032688Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-12-04T09:51:51.0032998Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-12-04T09:51:51.0033333Z  # We don't want the job to fail if the directory doesn't exist 2025-12-04T09:51:51.0033674Z  exit 0 2025-12-04T09:51:51.0033847Z fi 2025-12-04T09:51:51.0034010Z  2025-12-04T09:51:51.0034203Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-12-04T09:51:51.0034526Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-12-04T09:51:51.0034903Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-12-04T09:51:51.0035200Z  --metadata "${BENCHMARK_METADATA}" \ 2025-12-04T09:51:51.0035442Z  --runners "${RUNNER_INFO}" \ 2025-12-04T09:51:51.0035695Z  --dependencies "${DEPENDENCIES}" \ 2025-12-04T09:51:51.0035934Z  --dry-run 2025-12-04T09:51:51.0036122Z else 2025-12-04T09:51:51.0036378Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-12-04T09:51:51.0036739Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-12-04T09:51:51.0037022Z  --metadata "${BENCHMARK_METADATA}" \ 2025-12-04T09:51:51.0037258Z  --runners "${RUNNER_INFO}" \ 2025-12-04T09:51:51.0037506Z  --dependencies "${DEPENDENCIES}" 2025-12-04T09:51:51.0037727Z fi 2025-12-04T09:51:51.0042236Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:51.0042505Z env: 2025-12-04T09:51:51.0042683Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:51.0042890Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:51.0043213Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:51.0043556Z DEVICE_NAME: 2025-12-04T09:51:51.0043733Z DEVICE_TYPE: 2025-12-04T09:51:51.0043928Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-12-04T09:51:51.0044161Z DRY_RUN: false 2025-12-04T09:51:51.0045194Z BENCHMARK_METADATA: {"timestamp": 1764841910, "schema_version": "v3", "name": "periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32", "workflow_id": 19923066595, "run_attempt": 1, "job_id": 57118563290} 2025-12-04T09:51:51.0046417Z RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-39-147.ec2.internal"}, "name": "", "type": ""}] 2025-12-04T09:51:51.0046843Z DEPENDENCIES: {} 2025-12-04T09:51:51.0047018Z ##[endgroup] 2025-12-04T09:51:51.0071548Z + [[ -n '' ]] 2025-12-04T09:51:51.0072239Z + [[ ! -d test/test-reports ]] 2025-12-04T09:51:51.0072595Z + [[ false == \t\r\u\e ]] 2025-12-04T09:51:51.0074488Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1764841910, "schema_version": "v3", "name": "periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32", "workflow_id": 19923066595, "run_attempt": 1, "job_id": 57118563290}' --runners '[{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-39-147.ec2.internal"}, "name": "", "type": ""}]' --dependencies '{}' 2025-12-04T09:51:51.1275265Z INFO:root:Upload test/test-reports/inference_huggingface.json to s3://ossci-benchmarks/v3/pytorch/pytorch/19923066595/57118563290/inference_huggingface.json 2025-12-04T09:51:51.1556396Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role 2025-12-04T09:51:51.3654679Z ##[group]Run cat test/**/*_toprint.log || true 2025-12-04T09:51:51.3655029Z cat test/**/*_toprint.log || true 2025-12-04T09:51:51.3659684Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:51.3659942Z env: 2025-12-04T09:51:51.3660122Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:51.3660324Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:51.3660634Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:51.3660981Z DEVICE_NAME: 2025-12-04T09:51:51.3661159Z DEVICE_TYPE: 2025-12-04T09:51:51.3661334Z ##[endgroup] 2025-12-04T09:51:51.3740625Z cat: 'test/**/*_toprint.log': No such file or directory 2025-12-04T09:51:51.3790610Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-12-04T09:51:51.3790885Z kill "$MONITOR_SCRIPT_PID" 2025-12-04T09:51:51.3795636Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:51.3795898Z env: 2025-12-04T09:51:51.3796072Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:51.3796280Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:51.3796595Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:51.3796938Z DEVICE_NAME: 2025-12-04T09:51:51.3797113Z DEVICE_TYPE: 2025-12-04T09:51:51.3797293Z MONITOR_SCRIPT_PID: 48616 2025-12-04T09:51:51.3797483Z ##[endgroup] 2025-12-04T09:51:51.3818169Z /home/ec2-user/actions-runner/_work/_temp/9fa11430-8ff0-48ca-97c5-9b411017c899.sh: line 1: kill: (48616) - No such process 2025-12-04T09:51:51.3828100Z ##[error]Process completed with exit code 1. 2025-12-04T09:51:51.3919380Z Prepare all required actions 2025-12-04T09:51:51.3919772Z Getting action download info 2025-12-04T09:51:51.5795574Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-12-04T09:51:51.8126232Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-12-04T09:51:52.3227583Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-12-04T09:51:52.3227876Z with: 2025-12-04T09:51:52.3228192Z file-suffix: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290 2025-12-04T09:51:52.3228591Z s3-bucket: gha-artifacts 2025-12-04T09:51:52.3228787Z env: 2025-12-04T09:51:52.3228959Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:52.3229160Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:52.3229515Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:52.3229987Z DEVICE_NAME: 2025-12-04T09:51:52.3230198Z DEVICE_TYPE: 2025-12-04T09:51:52.3230371Z ##[endgroup] 2025-12-04T09:51:52.3249779Z ##[group]Run # Remove any previous test jsons if they exist 2025-12-04T09:51:52.3250109Z # Remove any previous test jsons if they exist 2025-12-04T09:51:52.3250383Z rm -f test-jsons-*.zip 2025-12-04T09:51:52.3250692Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-12-04T09:51:52.3255873Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:52.3256142Z env: 2025-12-04T09:51:52.3256329Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:52.3256533Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:52.3256853Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:52.3257201Z DEVICE_NAME: 2025-12-04T09:51:52.3257378Z DEVICE_TYPE: 2025-12-04T09:51:52.3257669Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290 2025-12-04T09:51:52.3258009Z ##[endgroup] 2025-12-04T09:51:52.3428263Z adding: test/test-reports/inference_huggingface.json (deflated 99%) 2025-12-04T09:51:52.3472071Z ##[group]Run # Remove any previous test reports if they exist 2025-12-04T09:51:52.3472414Z # Remove any previous test reports if they exist 2025-12-04T09:51:52.3472675Z rm -f test-reports-*.zip 2025-12-04T09:51:52.3473001Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-12-04T09:51:52.3477979Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:52.3478321Z env: 2025-12-04T09:51:52.3478502Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:52.3478710Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:52.3479031Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:52.3479375Z DEVICE_NAME: 2025-12-04T09:51:52.3479555Z DEVICE_TYPE: 2025-12-04T09:51:52.3479855Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290 2025-12-04T09:51:52.3480184Z ##[endgroup] 2025-12-04T09:51:52.3548240Z adding: test/test-reports/inference_huggingface.csv (deflated 63%) 2025-12-04T09:51:52.3548697Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 85%) 2025-12-04T09:51:52.3549202Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 65%) 2025-12-04T09:51:52.3574537Z ##[group]Run # Remove any previous usage logs if they exist 2025-12-04T09:51:52.3574858Z # Remove any previous usage logs if they exist 2025-12-04T09:51:52.3575128Z rm -f logs-*.zip 2025-12-04T09:51:52.3575382Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-12-04T09:51:52.3575724Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-12-04T09:51:52.3580070Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:52.3580337Z env: 2025-12-04T09:51:52.3580514Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:52.3580707Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:52.3581153Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:52.3581496Z DEVICE_NAME: 2025-12-04T09:51:52.3581661Z DEVICE_TYPE: 2025-12-04T09:51:52.3581959Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290 2025-12-04T09:51:52.3582276Z ##[endgroup] 2025-12-04T09:51:52.3622251Z adding: usage_log.txt (deflated 58%) 2025-12-04T09:51:52.3633520Z 2025-12-04T09:51:52.3634108Z zip error: Nothing to do! (logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290.zip) 2025-12-04T09:51:52.3651992Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-12-04T09:51:52.3652348Z # Remove any previous debugging artifacts if they exist 2025-12-04T09:51:52.3652610Z rm -f debug-*.zip 2025-12-04T09:51:52.3652813Z if [ -d 'test/debug' ]; then 2025-12-04T09:51:52.3653122Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-12-04T09:51:52.3653339Z fi 2025-12-04T09:51:52.3657252Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:52.3657487Z env: 2025-12-04T09:51:52.3657652Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:52.3657841Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:52.3658157Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:52.3658485Z DEVICE_NAME: 2025-12-04T09:51:52.3658660Z DEVICE_TYPE: 2025-12-04T09:51:52.3658958Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290 2025-12-04T09:51:52.3659276Z ##[endgroup] 2025-12-04T09:51:52.3725814Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:51:52.3726060Z with: 2025-12-04T09:51:52.3726250Z s3-bucket: gha-artifacts 2025-12-04T09:51:52.3726502Z s3-prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:51:52.3726753Z retention-days: 14 2025-12-04T09:51:52.3726962Z if-no-files-found: warn 2025-12-04T09:51:52.3727178Z path: test-jsons-*.zip 2025-12-04T09:51:52.3727372Z name: artifact 2025-12-04T09:51:52.3727557Z region: us-east-1 2025-12-04T09:51:52.3727736Z env: 2025-12-04T09:51:52.3727906Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:52.3728110Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:52.3728443Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:52.3728800Z DEVICE_NAME: 2025-12-04T09:51:52.3728975Z DEVICE_TYPE: 2025-12-04T09:51:52.3729151Z ##[endgroup] 2025-12-04T09:51:52.6445065Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T09:51:52.6449796Z With the provided path, there will be 1 file uploaded 2025-12-04T09:51:52.6452348Z Uploading to s3 prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:51:52.6478789Z Starting upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290.zip 2025-12-04T09:51:52.7660864Z Finished upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290.zip 2025-12-04T09:51:52.7813099Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:51:52.7813349Z with: 2025-12-04T09:51:52.7813533Z s3-bucket: gha-artifacts 2025-12-04T09:51:52.7813777Z s3-prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:51:52.7814017Z retention-days: 14 2025-12-04T09:51:52.7814206Z if-no-files-found: error 2025-12-04T09:51:52.7814416Z path: test-reports-*.zip 2025-12-04T09:51:52.7814605Z name: artifact 2025-12-04T09:51:52.7814784Z region: us-east-1 2025-12-04T09:51:52.7815008Z env: 2025-12-04T09:51:52.7815174Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:52.7815378Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:52.7815712Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:52.7816063Z DEVICE_NAME: 2025-12-04T09:51:52.7816238Z DEVICE_TYPE: 2025-12-04T09:51:52.7816413Z ##[endgroup] 2025-12-04T09:51:53.0820622Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T09:51:53.0825065Z With the provided path, there will be 1 file uploaded 2025-12-04T09:51:53.0825481Z Uploading to s3 prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:51:53.0851465Z Starting upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290.zip 2025-12-04T09:51:53.1990561Z Finished upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290.zip 2025-12-04T09:51:53.2133551Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:51:53.2133822Z with: 2025-12-04T09:51:53.2134009Z s3-bucket: gha-artifacts 2025-12-04T09:51:53.2134245Z s3-prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:51:53.2134496Z retention-days: 14 2025-12-04T09:51:53.2134689Z if-no-files-found: ignore 2025-12-04T09:51:53.2134886Z path: logs-*.zip 2025-12-04T09:51:53.2135060Z name: artifact 2025-12-04T09:51:53.2135234Z region: us-east-1 2025-12-04T09:51:53.2135397Z env: 2025-12-04T09:51:53.2135636Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:53.2135839Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:53.2136167Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:53.2136514Z DEVICE_NAME: 2025-12-04T09:51:53.2136694Z DEVICE_TYPE: 2025-12-04T09:51:53.2136861Z ##[endgroup] 2025-12-04T09:51:53.4713595Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T09:51:53.4718574Z With the provided path, there will be 1 file uploaded 2025-12-04T09:51:53.4719096Z Uploading to s3 prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:51:53.4746987Z Starting upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290.zip 2025-12-04T09:51:53.5862241Z Finished upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563290.zip 2025-12-04T09:51:53.6052507Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:51:53.6052748Z with: 2025-12-04T09:51:53.6052927Z s3-bucket: gha-artifacts 2025-12-04T09:51:53.6053182Z s3-prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:51:53.6053431Z retention-days: 14 2025-12-04T09:51:53.6053620Z if-no-files-found: ignore 2025-12-04T09:51:53.6053819Z path: debug-*.zip 2025-12-04T09:51:53.6053985Z name: artifact 2025-12-04T09:51:53.6054158Z region: us-east-1 2025-12-04T09:51:53.6054316Z env: 2025-12-04T09:51:53.6054463Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:53.6054651Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:53.6054947Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:53.6055373Z DEVICE_NAME: 2025-12-04T09:51:53.6055541Z DEVICE_TYPE: 2025-12-04T09:51:53.6055709Z ##[endgroup] 2025-12-04T09:51:53.8588540Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-12-04T09:51:53.8758454Z ##[group]Run # shellcheck disable=SC2156 2025-12-04T09:51:53.8758736Z # shellcheck disable=SC2156 2025-12-04T09:51:53.8759148Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-12-04T09:51:53.8764570Z shell: /usr/bin/bash -e {0} 2025-12-04T09:51:53.8764788Z env: 2025-12-04T09:51:53.8764970Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:53.8765170Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:53.8765501Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:53.8765850Z DEVICE_NAME: 2025-12-04T09:51:53.8766027Z DEVICE_TYPE: 2025-12-04T09:51:53.8766198Z ##[endgroup] 2025-12-04T09:51:54.0656020Z Prepare all required actions 2025-12-04T09:51:54.0656374Z Getting action download info 2025-12-04T09:51:54.2210983Z Download action repository 'actions/setup-python@v6' (SHA:83679a892e2d95755f2dac6acb0bfd1e9ac5d548) 2025-12-04T09:51:54.6353074Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-12-04T09:51:54.6353334Z with: 2025-12-04T09:51:54.6353564Z job_id: 57118563290 2025-12-04T09:51:54.6353987Z job_name: periodic-dynamo-benchmarks-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:51:54.6354428Z workflow_name: inductor-periodic 2025-12-04T09:51:54.6354651Z workflow_run_id: 19923066595 2025-12-04T09:51:54.6354858Z workflow_attempt: 1 2025-12-04T09:51:54.6355036Z env: 2025-12-04T09:51:54.6355197Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:54.6355397Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:54.6355713Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:54.6356057Z DEVICE_NAME: 2025-12-04T09:51:54.6356227Z DEVICE_TYPE: 2025-12-04T09:51:54.6356399Z ##[endgroup] 2025-12-04T09:51:54.6385120Z ##[group]Run actions/setup-python@v6 2025-12-04T09:51:54.6385385Z with: 2025-12-04T09:51:54.6385588Z python-version: 3.10 2025-12-04T09:51:54.6385798Z check-latest: false 2025-12-04T09:51:54.6386188Z token: *** 2025-12-04T09:51:54.6386412Z update-environment: true 2025-12-04T09:51:54.6386650Z allow-prereleases: false 2025-12-04T09:51:54.6386954Z freethreaded: false 2025-12-04T09:51:54.6387151Z env: 2025-12-04T09:51:54.6387329Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:54.6387533Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:54.6387892Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:54.6388283Z DEVICE_NAME: 2025-12-04T09:51:54.6388476Z DEVICE_TYPE: 2025-12-04T09:51:54.6388672Z ##[endgroup] 2025-12-04T09:51:54.7719870Z ##[group]Installed versions 2025-12-04T09:51:54.7726156Z Version 3.10 was not found in the local cache 2025-12-04T09:51:54.7877085Z (node:68373) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead. 2025-12-04T09:51:54.7877620Z (Use `node --trace-deprecation ...` to show where the warning was created) 2025-12-04T09:51:55.1081933Z ##[error]The version '3.10' with architecture 'x64' was not found for this operating system. The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json 2025-12-04T09:51:55.1218542Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-12-04T09:51:55.1218862Z with: 2025-12-04T09:51:55.1219037Z env: 2025-12-04T09:51:55.1219206Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:55.1219411Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:55.1219743Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:55.1220123Z DEVICE_NAME: 2025-12-04T09:51:55.1220318Z DEVICE_TYPE: 2025-12-04T09:51:55.1220584Z ##[endgroup] 2025-12-04T09:51:55.1232570Z ##[group]Run set -eou pipefail 2025-12-04T09:51:55.1232837Z set -eou pipefail 2025-12-04T09:51:55.1233030Z  2025-12-04T09:51:55.1233291Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-12-04T09:51:55.1233602Z for _ in $(seq 1440); do 2025-12-04T09:51:55.1233835Z  # Break if no ssh session exists anymore 2025-12-04T09:51:55.1234086Z  if [ "$(who)" = "" ]; then 2025-12-04T09:51:55.1234357Z  break 2025-12-04T09:51:55.1234568Z  fi 2025-12-04T09:51:55.1234735Z  echo "." 2025-12-04T09:51:55.1234908Z  sleep 5 2025-12-04T09:51:55.1235072Z done 2025-12-04T09:51:55.1240264Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:55.1240560Z env: 2025-12-04T09:51:55.1240869Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:55.1241102Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:55.1241470Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:55.1241874Z DEVICE_NAME: 2025-12-04T09:51:55.1242081Z DEVICE_TYPE: 2025-12-04T09:51:55.1242290Z ##[endgroup] 2025-12-04T09:51:55.1267300Z Holding runner for 2 hours until all ssh sessions have logged out 2025-12-04T09:51:55.1353388Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:51:55.1353771Z # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:51:55.1354092Z # shellcheck disable=SC2046 2025-12-04T09:51:55.1354337Z docker stop $(docker ps -q) || true 2025-12-04T09:51:55.1354581Z # Prune all of the docker images 2025-12-04T09:51:55.1354810Z docker system prune -af 2025-12-04T09:51:55.1359117Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:51:55.1359378Z env: 2025-12-04T09:51:55.1359548Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:51:55.1359744Z HAS_NVIDIA_GPU: false 2025-12-04T09:51:55.1360074Z DOCKER_CONTAINER_ID: b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:51:55.1360421Z DEVICE_NAME: 2025-12-04T09:51:55.1360893Z DEVICE_TYPE: 2025-12-04T09:51:55.1361086Z ##[endgroup] 2025-12-04T09:52:05.9565508Z b931eaab4b96 2025-12-04T09:52:06.3125118Z Deleted Containers: 2025-12-04T09:52:06.3127588Z b931eaab4b96ca3a8d55605c93add9370176425ea8dd751a39ab112c7dd402c8 2025-12-04T09:52:06.3128277Z 2025-12-04T09:52:14.0456951Z Deleted Images: 2025-12-04T09:52:14.0457643Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:52:14.0458531Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:b178ee928adfcab963ebbc6ea05ea1a0f0c605bb095e9076f010bf92d150869d 2025-12-04T09:52:14.0459162Z deleted: sha256:40bd34edc3497e05e2150804a15313235b6426bcc4b8c19c03c3daf5524be913 2025-12-04T09:52:14.0459561Z deleted: sha256:947006dce32e090f6b67d556f519398b957121651577f10819c7b179a5b6b938 2025-12-04T09:52:14.0459960Z deleted: sha256:dca16630fc5508e008012b760ac178fac5af74cb679706a3d9898100501d6f8e 2025-12-04T09:52:14.0460368Z deleted: sha256:a35e2e4cfaaa05e1f193fbe67c4151ded52737cb30c75429050cb7cd7777da23 2025-12-04T09:52:14.0460807Z deleted: sha256:d91e1a0a76e64abd7933fc58b8a553d76ac6cd82269c1bf701ade4e2c5ab58ff 2025-12-04T09:52:14.0461229Z deleted: sha256:f96fff40164ffa27ae535a4e5da2bbf2b18ec293366adc73519c830606f07baa 2025-12-04T09:52:14.0462132Z deleted: sha256:649089a3156b98c2cbf16b0af2b1ebdaa8644f9bf90534d9a1f7657b78b9f90e 2025-12-04T09:52:14.0462544Z deleted: sha256:aebabb7ea7535ad2aca44b23eb308b16d5de44f5b412574f94620d923436f42e 2025-12-04T09:52:14.0462940Z deleted: sha256:bcb08dc52ae55732fc33ab8c011347049a9279cc5f45a50d960c95d3a7cd5a3d 2025-12-04T09:52:14.0463328Z deleted: sha256:aa538cda5b22ea6e81fb69326637459a1356cce39b618657a075ce25d9124876 2025-12-04T09:52:14.0463726Z deleted: sha256:e28671ddaf02cbac7c1dcfdb2b08a40d3b7e2dab69c9a9e6c48ede3d8d550d66 2025-12-04T09:52:14.0464195Z deleted: sha256:ba797a4f759946d2fa7414073a5590cc12337aaa43197c35326ce456a3f84adf 2025-12-04T09:52:14.0464578Z deleted: sha256:25b28b66e2a53c3498a56aff9f543bf3a4641534c9e69741c673e9d018c652d8 2025-12-04T09:52:14.0464963Z deleted: sha256:860240198a7e7dcec697a58bcf979cfde64334033f2e97d4c9609277e051558c 2025-12-04T09:52:14.0465339Z deleted: sha256:39de54a9c331b8da952419605b26c35862b4cbb8c8a0967cc536a02b96eda874 2025-12-04T09:52:14.0465726Z deleted: sha256:6f78957316e0887707bc6534ea2aa43d1cd016130ec6a0d3148370b879696e1a 2025-12-04T09:52:14.0466156Z deleted: sha256:8656f7aa172ef2a37ec1524ac6f3ecd2b3fd9675a78a777b9a902d0f26047ea9 2025-12-04T09:52:14.0466606Z deleted: sha256:255344ff6c764e507ea2f21fbbdc6e0a1aaab0e5e0835935494c9df7d620e687 2025-12-04T09:52:14.0467035Z deleted: sha256:b6a1addb4370e1500d2fd65255b80922ad308c95a846a3460be730442d75fbe4 2025-12-04T09:52:14.0467444Z deleted: sha256:8dafa1eff3c9aada8b6725626ba0dd7de62719fb5ff307be3005cf2c6e72e2df 2025-12-04T09:52:14.0467886Z deleted: sha256:9e3fde92e8ff4f05cadac548766ef5ef9e8beb3c89cb48b4e2a17843ff60d649 2025-12-04T09:52:14.0468275Z deleted: sha256:7169a6f70d1f624733e52107b4913effe35e1e63c99388164b0afd2345658178 2025-12-04T09:52:14.0468657Z deleted: sha256:4c0d4c67766b690c1f1834310cd3898a900cd9742b7157de7e8f64d651600e35 2025-12-04T09:52:14.0469060Z deleted: sha256:7343a9c5c68976b0328278e691ddbcf26f619f989ca2a51933ac8bf637d1d69c 2025-12-04T09:52:14.0469490Z deleted: sha256:b790f29be6db3408a1a53fabdc69e76bb38cef221474a4f3aef1e233bb85d6d3 2025-12-04T09:52:14.0469902Z deleted: sha256:ed0831c232a6a410214e113695ab9288063623b523a6032a05b443fa3e9a68ae 2025-12-04T09:52:14.0470307Z deleted: sha256:554ed49d988a9f7a1381d193479542175c40d702a22d246ecef7958a8662a2eb 2025-12-04T09:52:14.0470750Z deleted: sha256:47b04d260bfb2b1f68a2f12aa70926ae2a2a53be6f4d8811d6fbe17fd7f13e17 2025-12-04T09:52:14.0471150Z deleted: sha256:6eb59e87109db04da60df057ff6b232045b6df4e2729ce7534a7a77fbe663761 2025-12-04T09:52:14.0471585Z deleted: sha256:0d9763e3d6e83eee92752c3a5dcfce74703dfa0aba42042761d7600edd95baa5 2025-12-04T09:52:14.0471977Z deleted: sha256:5811882cf3f14a50f47dc91a87d1be09daf3c20494aaf7e7343554066e9f90b7 2025-12-04T09:52:14.0472355Z deleted: sha256:4a6f4f545e9062949802a6c597609ef772633eae7d6925d41823d2b1a8991d4d 2025-12-04T09:52:14.0472757Z deleted: sha256:69a716cb9dc95d7d58d1416f4d127e49885c5d96882dc593686a6eb97200582a 2025-12-04T09:52:14.0473137Z deleted: sha256:0542610effe51f46308d4570438af559e4f6a263f38714ea04c78e4661d1909a 2025-12-04T09:52:14.0473578Z deleted: sha256:86dcdb716a38a00a400b483f42e4ac58714a70e89f5671b50ca5c45d8f577217 2025-12-04T09:52:14.0474009Z deleted: sha256:a9aaf6de8156c1b1a8ece816adbc7d2ed6edabb95e36732737ca293b26619b40 2025-12-04T09:52:14.0474431Z deleted: sha256:57ce7fcbaaa86795f8f7e6290c0aaa8622e13c9c82191d34952c0a4d2e203326 2025-12-04T09:52:14.0474854Z deleted: sha256:a90408b5f4c06a3ba881facc34fa55beb4ae4e51823e7f8d933e611f68465f4a 2025-12-04T09:52:14.0475281Z deleted: sha256:1ac1aab20bc98902f03ee9c657ff435af431f40a81ac9cebf1cd91d9041bc6b1 2025-12-04T09:52:14.0475719Z deleted: sha256:89e6c1c76b21fbe98c153a32627fa1b2337a8caa744b09b28253b67cb84012cb 2025-12-04T09:52:14.0476135Z deleted: sha256:3765d58e17aa81539a8e1eb3cb45914b48f8fb53f7ac12c1bf06c9390e274a93 2025-12-04T09:52:14.0476551Z deleted: sha256:6056a51594e8137d3425100cc233c8ab0f09dded2dc491ba083e8d7c0e1f1143 2025-12-04T09:52:14.0477007Z deleted: sha256:43a55512a68fc9eb9c622a73682dbf9f3c83f1b499287d9e12151f559edabe85 2025-12-04T09:52:14.0477485Z deleted: sha256:a5ad8e88a7e08768a61feeb801fb5a4c5041587feb2a2c77c35b285e869c25c9 2025-12-04T09:52:14.0480498Z deleted: sha256:55fd389078a5792f567a3592821e38fcfa0b0e114fa43248c4528c4d85cf6459 2025-12-04T09:52:14.0481117Z deleted: sha256:144567b58c97f1b5a75d54c3a1514a26ad5ed2dc7988e166316e324a76e76aa6 2025-12-04T09:52:14.0481544Z deleted: sha256:049fbb3123a92952eba68198938b7b29203fcb3d74dfc74e67f9929999576f19 2025-12-04T09:52:14.0481957Z deleted: sha256:22607d21e148d19a45f3f7afe927a15041b9a5663430f5580cc993c4bf9dd7dc 2025-12-04T09:52:14.0482419Z deleted: sha256:63a29311926064245f6d4a4661513586c8179b1cb460ceb91184c0d1114a742d 2025-12-04T09:52:14.0482823Z deleted: sha256:0072101bc0e6bbe2457f4a2d0de953ae643fc2aa6adc69bb59da9cbf6006c8a9 2025-12-04T09:52:14.0483245Z deleted: sha256:35c8f84104fcf39bc1be4682f46dc8b04cb1c5e4c26f66aac039cb42a3f08128 2025-12-04T09:52:14.0483663Z deleted: sha256:c6da65c085b8e72b2e240a3a7d79c4aa166c247737f5916587e51d40acff1ab8 2025-12-04T09:52:14.0484071Z deleted: sha256:b8b5ced5e8550748e26855aad0bb30864c4c435c64694097e63a9a686ef90ba0 2025-12-04T09:52:14.0484472Z deleted: sha256:9905f824759cfae96d29353305bc8a6130a7316db29dfc3efeb8ba9c7cbb5172 2025-12-04T09:52:14.0484888Z deleted: sha256:2b93a73a0fc57cdcfab2a04a30c9b313d84e5e447d076575212f198508242a00 2025-12-04T09:52:14.0485289Z deleted: sha256:baa73842b47530e5d1421010d895ed9059a567516a7e85b10943a6b5fe6fbe41 2025-12-04T09:52:14.0485685Z deleted: sha256:db79627a360d2322d58fb57c98268676011788961d4682ba034f17c9f5fdd855 2025-12-04T09:52:14.0486108Z deleted: sha256:8439f69d5eab9d42b15df43b938f56c690b18cd5e3884f4b80ddc9edf9e43822 2025-12-04T09:52:14.0486538Z deleted: sha256:e95ccdb10d43bbb6f2d6416bca8e5ad7d76a39985431aa5e40ff192ebeb52883 2025-12-04T09:52:14.0486968Z deleted: sha256:d878081c591fd25e8e494f4565e67f341cb3c459b21596ad8e8a707ff9ca63e6 2025-12-04T09:52:14.0487372Z deleted: sha256:2d594947cf73e58a20e36d52acf22f3d31b7a43291d7017c1ac71ea38c4aa9d1 2025-12-04T09:52:14.0487787Z deleted: sha256:933d315b5f20ac941653d01f2b896cc7710564c9b9c4bf9fd8e850115180c6f9 2025-12-04T09:52:14.0488193Z deleted: sha256:0edebbabed274b9e777a5933aaa21ed58569a412c97423c365eff4645e773981 2025-12-04T09:52:14.0488609Z deleted: sha256:744e1cebe181659b0a93ac857d4c16af4c2ce29f3227368eacd7ef18b69e06a7 2025-12-04T09:52:14.0489023Z deleted: sha256:71393f74ab60b183866b9f5995f85aaa65613968c611c8dc15d043befbac662b 2025-12-04T09:52:14.0489427Z deleted: sha256:741a07a29d03bdecf1fb4dfc8f1654a136064f14e5ded89a1497f476ff24623d 2025-12-04T09:52:14.0489834Z deleted: sha256:83df0b5b82e0926d6046012c3b5098dae9aa37177323a33d6eda09e392844b86 2025-12-04T09:52:14.0490238Z deleted: sha256:65b3d6509a37164c67b36965f4ca77ccb1a8e62bdd8f058e2d0886c82ba0940f 2025-12-04T09:52:14.0490659Z deleted: sha256:a38d6cb5dacc51c3e614f4ed42c05cdc53fc489f6e3dafc3b7d0eb39d3250908 2025-12-04T09:52:14.0491079Z deleted: sha256:c3de4f46ac5273a8218c6c5325247b0bd17f12037cfb49f5101a2b5085746704 2025-12-04T09:52:14.0491492Z deleted: sha256:7f3448d255710b28f79767ded93dcf11e0c1a941c7579d8752f4f0bc589aa0b5 2025-12-04T09:52:14.0491930Z deleted: sha256:fd33fe89098b4a8897870a9a316cd91868ab19f42f800d5b512e679356791e62 2025-12-04T09:52:14.0492327Z deleted: sha256:df2c12f62f5ceb2670bf7cb8aa53f268d11dbbfbf25f4b4d567673702cc3fc30 2025-12-04T09:52:14.0492741Z deleted: sha256:a3d7900a6180e4612708429f9c0edba3e999bb43fd67fd3406ae088ea7016a47 2025-12-04T09:52:14.0493156Z deleted: sha256:f592639a1c1bf8da30a1c21866ee22e76b33ae029ee236c7fd20b1ecc4716775 2025-12-04T09:52:14.0493580Z deleted: sha256:e11e4a9756b2bd4a6e62cde476dc36c6528d71a74e8e7570b271784bc494ab40 2025-12-04T09:52:14.0493996Z deleted: sha256:4c03ab513fc4cf698dd61fc37312e57abf7c4e44d4d4ddab126f38804fddec75 2025-12-04T09:52:14.0494426Z deleted: sha256:b4d3d8b5232e0bac87fa78b9c77997cbe8798faffce2aac259b820ab387ce4ad 2025-12-04T09:52:14.0494855Z deleted: sha256:dfda7f6e86c7217a3eb1fe089352248b5d84ca4f62734abce70beb4a6fc90779 2025-12-04T09:52:14.0495273Z deleted: sha256:00f41b846c66bc26a6381fe3663c611a2f539ae3b7e46e007436afd3335a7517 2025-12-04T09:52:14.0495736Z deleted: sha256:0f35c2638249a4525b29668586b079b4ee01f73bbe33732c9da7cfa7b6afb480 2025-12-04T09:52:14.0496152Z deleted: sha256:7441da072ac3269ddb67441b265fdf82904871335b5c19bb381d3622a95b7745 2025-12-04T09:52:14.0496568Z deleted: sha256:ba70851a3aedb994e8d7f8657f8f108bd90a19272843f389e6a8af48a61db720 2025-12-04T09:52:14.0496989Z deleted: sha256:1984ce6f5cabbd8c86386a8d9b5365923b92bc7ccb9aace9b4fe9570213bbbe1 2025-12-04T09:52:14.0497416Z deleted: sha256:73974f74b436f39a2fdb6461b1e3f7c3e41c73325776fa71d16b942a5b4a365b 2025-12-04T09:52:14.0497691Z 2025-12-04T09:52:14.0497801Z Total reclaimed space: 53.33GB 2025-12-04T09:52:14.0559699Z Post job cleanup. 2025-12-04T09:52:14.0589505Z Post job cleanup. 2025-12-04T09:52:14.1821799Z (node:68531) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead. 2025-12-04T09:52:14.1822418Z (Use `node --trace-deprecation ...` to show where the warning was created) 2025-12-04T09:52:14.1940950Z Post job cleanup. 2025-12-04T09:52:14.1975264Z Post job cleanup. 2025-12-04T09:52:14.2888327Z [command]/usr/bin/git version 2025-12-04T09:52:14.2922351Z git version 2.50.1 2025-12-04T09:52:14.2961621Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/b1edd014-f3dc-4b56-9a02-efcd8c2369ff/.gitconfig' 2025-12-04T09:52:14.2982056Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/b1edd014-f3dc-4b56-9a02-efcd8c2369ff' before making global git config changes 2025-12-04T09:52:14.2987306Z Adding repository directory to the temporary git global config as a safe directory 2025-12-04T09:52:14.2992051Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:52:14.3030978Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-12-04T09:52:14.3066438Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-12-04T09:52:14.3381573Z Entering 'android/libs/fbjni' 2025-12-04T09:52:14.3436441Z Entering 'third_party/FP16' 2025-12-04T09:52:14.3495372Z Entering 'third_party/FXdiv' 2025-12-04T09:52:14.3548187Z Entering 'third_party/NNPACK' 2025-12-04T09:52:14.3601691Z Entering 'third_party/NVTX' 2025-12-04T09:52:14.3655991Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:52:14.3706732Z Entering 'third_party/XNNPACK' 2025-12-04T09:52:14.3774231Z Entering 'third_party/aiter' 2025-12-04T09:52:14.3823791Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:52:14.3897254Z Entering 'third_party/benchmark' 2025-12-04T09:52:14.3943477Z Entering 'third_party/composable_kernel' 2025-12-04T09:52:14.4015505Z Entering 'third_party/cpp-httplib' 2025-12-04T09:52:14.4062943Z Entering 'third_party/cpuinfo' 2025-12-04T09:52:14.4116184Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:52:14.4171331Z Entering 'third_party/cutlass' 2025-12-04T09:52:14.4233800Z Entering 'third_party/fbgemm' 2025-12-04T09:52:14.4295233Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:52:14.4352189Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:52:14.4412438Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:52:14.4464427Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:52:14.4522157Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:52:14.4577498Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:52:14.4631263Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:52:14.4690327Z Entering 'third_party/flash-attention' 2025-12-04T09:52:14.4744682Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:52:14.4806819Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:52:14.4865278Z Entering 'third_party/flatbuffers' 2025-12-04T09:52:14.4923080Z Entering 'third_party/fmt' 2025-12-04T09:52:14.4980161Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:52:14.5032195Z Entering 'third_party/gloo' 2025-12-04T09:52:14.5090624Z Entering 'third_party/googletest' 2025-12-04T09:52:14.5139271Z Entering 'third_party/ideep' 2025-12-04T09:52:14.5193654Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:52:14.5254220Z Entering 'third_party/ittapi' 2025-12-04T09:52:14.5306788Z Entering 'third_party/kineto' 2025-12-04T09:52:14.5365631Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:52:14.5416985Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:52:14.5472492Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:52:14.5528900Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:52:14.5587096Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:52:14.5642476Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:52:14.5710187Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:52:14.5762523Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:52:14.5819512Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:52:14.5881523Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:52:14.5934942Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:52:14.5987859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:52:14.6043319Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:52:14.6108831Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:52:14.6159679Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:52:14.6215844Z Entering 'third_party/kleidiai' 2025-12-04T09:52:14.6278817Z Entering 'third_party/mimalloc' 2025-12-04T09:52:14.6334707Z Entering 'third_party/nlohmann' 2025-12-04T09:52:14.6398465Z Entering 'third_party/onnx' 2025-12-04T09:52:14.6465164Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:52:14.6522864Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:52:14.6579253Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:52:14.6634457Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:52:14.6691915Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:52:14.6747627Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:52:14.6811976Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:52:14.6867504Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:52:14.6923795Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:52:14.6978666Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:52:14.7032175Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:52:14.7094166Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:52:14.7161979Z Entering 'third_party/pocketfft' 2025-12-04T09:52:14.7219783Z Entering 'third_party/protobuf' 2025-12-04T09:52:14.7277688Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:52:14.7336875Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:52:14.7398947Z Entering 'third_party/psimd' 2025-12-04T09:52:14.7459576Z Entering 'third_party/pthreadpool' 2025-12-04T09:52:14.7507596Z Entering 'third_party/pybind11' 2025-12-04T09:52:14.7566324Z Entering 'third_party/python-peachpy' 2025-12-04T09:52:14.7622460Z Entering 'third_party/sleef' 2025-12-04T09:52:14.7673537Z Entering 'third_party/tensorpipe' 2025-12-04T09:52:14.7731015Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:52:14.7780625Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:52:14.7833165Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:52:14.7891182Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:52:14.7938138Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:52:14.8021202Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-12-04T09:52:14.8041989Z http.https://github.com/.extraheader 2025-12-04T09:52:14.8054215Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-12-04T09:52:14.8079646Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-12-04T09:52:14.8396438Z Entering 'android/libs/fbjni' 2025-12-04T09:52:14.8433518Z http.https://github.com/.extraheader 2025-12-04T09:52:14.8468751Z Entering 'third_party/FP16' 2025-12-04T09:52:14.8505606Z http.https://github.com/.extraheader 2025-12-04T09:52:14.8536166Z Entering 'third_party/FXdiv' 2025-12-04T09:52:14.8572622Z http.https://github.com/.extraheader 2025-12-04T09:52:14.8610327Z Entering 'third_party/NNPACK' 2025-12-04T09:52:14.8651933Z http.https://github.com/.extraheader 2025-12-04T09:52:14.8692197Z Entering 'third_party/NVTX' 2025-12-04T09:52:14.8728218Z http.https://github.com/.extraheader 2025-12-04T09:52:14.8764213Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:52:14.8798728Z http.https://github.com/.extraheader 2025-12-04T09:52:14.8834705Z Entering 'third_party/XNNPACK' 2025-12-04T09:52:14.8873459Z http.https://github.com/.extraheader 2025-12-04T09:52:14.8919546Z Entering 'third_party/aiter' 2025-12-04T09:52:14.8960035Z http.https://github.com/.extraheader 2025-12-04T09:52:14.8992659Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:52:14.9029609Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9073344Z Entering 'third_party/benchmark' 2025-12-04T09:52:14.9106465Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9145275Z Entering 'third_party/composable_kernel' 2025-12-04T09:52:14.9181443Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9221807Z Entering 'third_party/cpp-httplib' 2025-12-04T09:52:14.9264413Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9296045Z Entering 'third_party/cpuinfo' 2025-12-04T09:52:14.9333541Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9374321Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:52:14.9403950Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9439781Z Entering 'third_party/cutlass' 2025-12-04T09:52:14.9481273Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9525253Z Entering 'third_party/fbgemm' 2025-12-04T09:52:14.9561302Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9599473Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:52:14.9635438Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9676694Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:52:14.9715543Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9750673Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:52:14.9787784Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9820963Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:52:14.9861516Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9902333Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:52:14.9938175Z http.https://github.com/.extraheader 2025-12-04T09:52:14.9970240Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:52:15.0006271Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0035670Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:52:15.0074509Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0111793Z Entering 'third_party/flash-attention' 2025-12-04T09:52:15.0148284Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0196008Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:52:15.0226049Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0266768Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:52:15.0306862Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0345492Z Entering 'third_party/flatbuffers' 2025-12-04T09:52:15.0381992Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0419440Z Entering 'third_party/fmt' 2025-12-04T09:52:15.0461641Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0493938Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:52:15.0529578Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0566548Z Entering 'third_party/gloo' 2025-12-04T09:52:15.0597902Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0631284Z Entering 'third_party/googletest' 2025-12-04T09:52:15.0669372Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0710721Z Entering 'third_party/ideep' 2025-12-04T09:52:15.0741924Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0776917Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:52:15.0815229Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0860336Z Entering 'third_party/ittapi' 2025-12-04T09:52:15.0892074Z http.https://github.com/.extraheader 2025-12-04T09:52:15.0928689Z Entering 'third_party/kineto' 2025-12-04T09:52:15.0964565Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1006972Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:52:15.1044611Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1076365Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:52:15.1116085Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1148480Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:52:15.1181349Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1215801Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:52:15.1250776Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1286100Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:52:15.1317019Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1355502Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:52:15.1390818Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1426310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:52:15.1463041Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1493823Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:52:15.1528941Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1560760Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:52:15.1595109Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1633635Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:52:15.1668065Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1699766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:52:15.1732908Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1770823Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:52:15.1807573Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1847568Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:52:15.1879728Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1929540Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:52:15.1960778Z http.https://github.com/.extraheader 2025-12-04T09:52:15.1995224Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:52:15.2030796Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2070732Z Entering 'third_party/kleidiai' 2025-12-04T09:52:15.2102694Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2138983Z Entering 'third_party/mimalloc' 2025-12-04T09:52:15.2175373Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2212337Z Entering 'third_party/nlohmann' 2025-12-04T09:52:15.2247495Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2285846Z Entering 'third_party/onnx' 2025-12-04T09:52:15.2317393Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2366186Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:52:15.2403425Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2439478Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:52:15.2475393Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2514576Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:52:15.2548526Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2593201Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:52:15.2626125Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2667094Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:52:15.2697709Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2733239Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:52:15.2766319Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2797404Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:52:15.2831895Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2875385Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:52:15.2907540Z http.https://github.com/.extraheader 2025-12-04T09:52:15.2940594Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:52:15.2978616Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3012131Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:52:15.3045339Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3078103Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:52:15.3113754Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3154592Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:52:15.3190339Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3241670Z Entering 'third_party/pocketfft' 2025-12-04T09:52:15.3279085Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3312328Z Entering 'third_party/protobuf' 2025-12-04T09:52:15.3348452Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3393280Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:52:15.3431026Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3461666Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:52:15.3495969Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3531287Z Entering 'third_party/psimd' 2025-12-04T09:52:15.3569755Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3608485Z Entering 'third_party/pthreadpool' 2025-12-04T09:52:15.3642391Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3679582Z Entering 'third_party/pybind11' 2025-12-04T09:52:15.3715680Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3755835Z Entering 'third_party/python-peachpy' 2025-12-04T09:52:15.3790799Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3827377Z Entering 'third_party/sleef' 2025-12-04T09:52:15.3860994Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3901705Z Entering 'third_party/tensorpipe' 2025-12-04T09:52:15.3932399Z http.https://github.com/.extraheader 2025-12-04T09:52:15.3969791Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:52:15.3997509Z http.https://github.com/.extraheader 2025-12-04T09:52:15.4034652Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:52:15.4071202Z http.https://github.com/.extraheader 2025-12-04T09:52:15.4098814Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:52:15.4139064Z http.https://github.com/.extraheader 2025-12-04T09:52:15.4179191Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:52:15.4208879Z http.https://github.com/.extraheader 2025-12-04T09:52:15.4238462Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:52:15.4280508Z http.https://github.com/.extraheader 2025-12-04T09:52:15.4337220Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.4387796Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url 2025-12-04T09:52:15.4694394Z Entering 'android/libs/fbjni' 2025-12-04T09:52:15.4717038Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-12-04T09:52:15.4733018Z Entering 'third_party/FP16' 2025-12-04T09:52:15.4762308Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-12-04T09:52:15.4780985Z Entering 'third_party/FXdiv' 2025-12-04T09:52:15.4802832Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-12-04T09:52:15.4813949Z Entering 'third_party/NNPACK' 2025-12-04T09:52:15.4837266Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-12-04T09:52:15.4863349Z Entering 'third_party/NVTX' 2025-12-04T09:52:15.4881187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-12-04T09:52:15.4899813Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:52:15.4920457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-12-04T09:52:15.4932666Z Entering 'third_party/XNNPACK' 2025-12-04T09:52:15.4958480Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-12-04T09:52:15.4989655Z Entering 'third_party/aiter' 2025-12-04T09:52:15.5011251Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-12-04T09:52:15.5026724Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:52:15.5054885Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-12-04T09:52:15.5077942Z Entering 'third_party/benchmark' 2025-12-04T09:52:15.5098964Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:52:15.5117829Z Entering 'third_party/composable_kernel' 2025-12-04T09:52:15.5134798Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-12-04T09:52:15.5161709Z Entering 'third_party/cpp-httplib' 2025-12-04T09:52:15.5182819Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-12-04T09:52:15.5199585Z Entering 'third_party/cpuinfo' 2025-12-04T09:52:15.5222993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-12-04T09:52:15.5237476Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:52:15.5260951Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-12-04T09:52:15.5280544Z Entering 'third_party/cutlass' 2025-12-04T09:52:15.5309649Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-12-04T09:52:15.5328549Z Entering 'third_party/fbgemm' 2025-12-04T09:52:15.5358262Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-12-04T09:52:15.5371964Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:52:15.5398563Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-12-04T09:52:15.5416577Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:52:15.5435406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-12-04T09:52:15.5457119Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:52:15.5481098Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-12-04T09:52:15.5498595Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:52:15.5521330Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-12-04T09:52:15.5545832Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:52:15.5570037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-12-04T09:52:15.5590938Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:52:15.5617061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-12-04T09:52:15.5632150Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:52:15.5657934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-12-04T09:52:15.5678765Z Entering 'third_party/flash-attention' 2025-12-04T09:52:15.5701244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-12-04T09:52:15.5717581Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:52:15.5741983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-12-04T09:52:15.5762069Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:52:15.5786661Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-12-04T09:52:15.5811885Z Entering 'third_party/flatbuffers' 2025-12-04T09:52:15.5830321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-12-04T09:52:15.5848782Z Entering 'third_party/fmt' 2025-12-04T09:52:15.5874279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:52:15.5892454Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:52:15.5912180Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-12-04T09:52:15.5926564Z Entering 'third_party/gloo' 2025-12-04T09:52:15.5957679Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-12-04T09:52:15.5979159Z Entering 'third_party/googletest' 2025-12-04T09:52:15.5999083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:52:15.6015654Z Entering 'third_party/ideep' 2025-12-04T09:52:15.6039653Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-12-04T09:52:15.6062648Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:52:15.6079872Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-12-04T09:52:15.6105124Z Entering 'third_party/ittapi' 2025-12-04T09:52:15.6126597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-12-04T09:52:15.6143765Z Entering 'third_party/kineto' 2025-12-04T09:52:15.6171642Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-12-04T09:52:15.6189882Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:52:15.6214051Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-12-04T09:52:15.6222385Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:52:15.6248430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-12-04T09:52:15.6264885Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:52:15.6287522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-12-04T09:52:15.6307376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:52:15.6330557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:52:15.6349605Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:52:15.6380066Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-12-04T09:52:15.6394824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:52:15.6415534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-12-04T09:52:15.6430789Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:52:15.6455618Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-12-04T09:52:15.6474761Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:52:15.6497906Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:52:15.6514977Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:52:15.6535269Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-12-04T09:52:15.6556501Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:52:15.6577701Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-12-04T09:52:15.6602240Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:52:15.6626719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:52:15.6640118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:52:15.6668777Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:52:15.6687339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:52:15.6709299Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:52:15.6731299Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:52:15.6759047Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-12-04T09:52:15.6774437Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:52:15.6797631Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-12-04T09:52:15.6822328Z Entering 'third_party/kleidiai' 2025-12-04T09:52:15.6841202Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-12-04T09:52:15.6859911Z Entering 'third_party/mimalloc' 2025-12-04T09:52:15.6885037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-12-04T09:52:15.6901372Z Entering 'third_party/nlohmann' 2025-12-04T09:52:15.6929665Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-12-04T09:52:15.6946770Z Entering 'third_party/onnx' 2025-12-04T09:52:15.6973036Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-12-04T09:52:15.7002755Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:52:15.7028222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:52:15.7047472Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:52:15.7072238Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-12-04T09:52:15.7087220Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:52:15.7114156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:52:15.7131436Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:52:15.7153940Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:52:15.7173894Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:52:15.7194920Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-12-04T09:52:15.7211757Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:52:15.7236271Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-12-04T09:52:15.7255099Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:52:15.7279228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-12-04T09:52:15.7296338Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:52:15.7321192Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-12-04T09:52:15.7335994Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:52:15.7359992Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:52:15.7379524Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:52:15.7402564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:52:15.7422001Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:52:15.7442999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:52:15.7464666Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:52:15.7485602Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-12-04T09:52:15.7520880Z Entering 'third_party/pocketfft' 2025-12-04T09:52:15.7543606Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-12-04T09:52:15.7560181Z Entering 'third_party/protobuf' 2025-12-04T09:52:15.7584797Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-12-04T09:52:15.7603398Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:52:15.7627715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:52:15.7644100Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:52:15.7670875Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:52:15.7687031Z Entering 'third_party/psimd' 2025-12-04T09:52:15.7709334Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-12-04T09:52:15.7727882Z Entering 'third_party/pthreadpool' 2025-12-04T09:52:15.7750380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-12-04T09:52:15.7768940Z Entering 'third_party/pybind11' 2025-12-04T09:52:15.7794846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:52:15.7812984Z Entering 'third_party/python-peachpy' 2025-12-04T09:52:15.7838559Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-12-04T09:52:15.7853433Z Entering 'third_party/sleef' 2025-12-04T09:52:15.7877237Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-12-04T09:52:15.7894563Z Entering 'third_party/tensorpipe' 2025-12-04T09:52:15.7918729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-12-04T09:52:15.7935239Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:52:15.7959315Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:52:15.7978572Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:52:15.8001403Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-12-04T09:52:15.8018455Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:52:15.8042001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-12-04T09:52:15.8061010Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:52:15.8085648Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:52:15.8099240Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:52:15.8121481Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-12-04T09:52:15.8161496Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8197805Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8223282Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8251273Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8286186Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8309764Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8333765Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8358033Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8385932Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8422152Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8453164Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8479825Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8506141Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8529319Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8556484Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8585254Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8615346Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8640319Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8665884Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8694098Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8719322Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8745690Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8780114Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8802661Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8837228Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8858072Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8886073Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8915043Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8942614Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8969780Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.8996591Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9020906Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9044858Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9082066Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9116887Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9145847Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9180272Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9203276Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9232450Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9266149Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9302947Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9334391Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9358687Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9388249Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9417833Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9443385Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9477711Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9503112Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9528761Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9555166Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9592810Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9621077Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9637430Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9660259Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9695183Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9725353Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9749330Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9782185Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9798630Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9828561Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9854348Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9882394Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9921159Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9944616Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:15.9984378Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0010509Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0033364Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0061575Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0089937Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0112165Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0138582Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0161590Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0199359Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0221266Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0254059Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0276812Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0308510Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0329858Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0358529Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0393187Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0419192Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:52:16.0531714Z A job completed hook has been configured by the self-hosted runner administrator 2025-12-04T09:52:16.0546043Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-12-04T09:52:16.0550375Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:52:16.0550670Z ##[endgroup] 2025-12-04T09:52:16.0638057Z [!ALERT!] Swap in detected! [!ALERT!] 2025-12-04T09:52:25.0099993Z [!ALERT!] Swap out detected [!ALERT!] 2025-12-04T09:52:39.6706752Z Cleaning up orphan processes